So I have tried taping the excess elastic in the back. I did not like it because it snagged my FOE. Then I tried the zip ties. They worked great, but didn't work so great at shows when people wanted to try them on.
So the other day I was browsing through Joannes and in the wedding section I saw the silver twist ties and came up with this:
It works great!