After sewing the two portions together, flip the hat. The folded flap should be on the outside now.
Cut about a 6 inch long piece of elastic band. Start with either the left or right side of the thick portion of the hat. About 4 inches from the edge, sew the elastic down on the inside of the flap. Sew down the flap to the fold and along the edge to the other side. Close up the gap by sewing down the other end of elastic 4 inches from the other side. After releasing the hat, it should scrunch together.
Note: If you feel the hat is still too loose, you can either shorten the length of the elastic or sew further out from the middle of the hat.