Here's what to do about that! Make a fitted hat with a fur or velvet inside. It's about the fastest hat there is, much faster than making a Hex Hat or even knitting a toque.
Step 1: Cut the Sides and Crown
Measure your head at its largest circumference. We will call this measurement H. From each of your two fabrics, cut a rectangle that is H plus 1.5 inches long, and 6 or 7 inches wide (depending on how big your head is). These will be the sides of the hat.
For the crown, cut an oval shape that's H + 2 inches around, or thereabouts (don't go much smaller than this). I made a paper pattern by folding a piece of paper in quarters, then cutting a line about (H + 2) / 4 inches long (see pic). Cut this out of your outer fabric.
Step 2: Construct the Sides
Turn one right-side out and place it inside the other so the right sides are together. It's tidy but not essential to match up the seams. Pin if you like, and stitch around the long edge.
Turn this right-side out now, so the seam is folded to the inside. I find it helpful to stitch the other long edge together at this point. This time the wrong sides are together.
Step 3: Attach the Crown
Find, and place pins to mark, the quarter distance around both edges. For the side piece, place a pin opposite to the seam, then fold so these two points meet and place a pin at the two fold points.
For the crown, fold the piece in half along each of the two axes in turn, and pin at the folds.
Now pin the two sections together at each of the marked quarter points. The sides should be inside out (soft fabric on the outside) and the crown should be upside down.
With the hat sides underneath and the crown on top, sew along the edge on approximately the line you sewed the two sides together along. Make sure the pieces remain matched up at the pins, and be careful not to get a fold of the piece underneath caught in the stitching. You may have to make a smaller or larger seam allowance on the crown depending on how well the lengths match up, or you may have to stretch the diagonal bits a little. All of this is fine, it won't show on the outside.
Step 4: Edgestitch and Turn
Turn the hat right side out and try it on!