I solved this with a helmet with integrated speakers (you can get those pretty cheap nowdays) and a separate Griffin microphone with an answer button. The mic is connected to the phone and the helmet speakers to a jack in the microphone. The microphone is attached (with safety pins) inside the jacket collar where it's protected from snow and wind but still catches the voice perfectly.
To solve the problem of being able to answer the phone with gloves on I made a simple switch handle to the jacket: I punched a hole through the collar right behind the mic, strengthened it with an eyelet and inserted a thin cord loop through it. The cord goes around the mic and when tightened it presses the answer button on the side of the mic.
Now, whenever the phone rings, simple pull from the cord gets you connected. Also, while listening to music the same gesture can be used to play, pause and skip. Works like a charm!