The complication: I think it's made of wool and maybe cardboard so washing might wreck it.
The solution: Baking soda (sodium bicarbonate) It's really amazing stuff. Keep some of it handy in case you spill sulfuric acid all over yourself.
Attention MICROBIOLOGISTS: Does NaHCO3 kill mold or yeast or anything else? Does this have anything to do with changes in pH or being a buffer?
Also there are other ways of doing this. Some suggest spraying on clear alcohol like vodka and gin. Apple cider vinegar would probably work nicely too, it has anti-fungal properties and works on athlete's foot.
Please feel free to share your own methods in the comments below.
Step 1: Do It and Bag It
Sprinkle lots of baking soda on all the garment's surfaces especially the seams and little hidden corners.
Put your garment in a bag to reduce spilled/lost baking soda. This way you can get all wild with your application without making a mess.
Rub it in good so the powder goes deep into the weave.
Step 2: Seal It Up
Step 3: Wait
In the mean time here are things the internetz cough up when one googles "wait".
Step 4: Open and Dust Off
lightly wipe it with a damp cloth
damp cloth again
As an experiment I left a little bit of baking soda in the hidden crevices as a prophylactic. Hopefully this won't harm the fabric over time.
Step 5: The Stink Is Gone
Actually this hat now smells like it did before it was musty. Success!
Step 6: Rock It!
Put it on and go paint the town brown.