Step 1: A brief history/physics/textiles lesson
Normally clothes keep you warm by trapping an insulating layer of air next to your skin- this is why many layers are better for warmth and the best undergarments (thermals or old-fashioned string vests etc) are actually mostly made of air. In windy conditions (for instance, sitting on a bike at 55mph), the wind blows straight through your clothes and removes this insulating layer, which is why you don't see many Harley owners wearing cardigans.
For warmth in cold and windy conditions, therefore, the best combination is an insulating under layer with a windproof layer over the top. Think about early aviators wearing sheepskin-lined leather jackets, or arctic-dwelling peoples wearing furs with the fur on the inside. Insulation underneath, windproof layer on top will be a recurring theme in this Instructable.
Step 2: Hot headed
On a motorbike this isn't much of an issue for me as my head is entirely encased in a windproof and insulating full face helmet, but cyclists need to consider their headwear carefully- a large proportion of the body's heat is lost through the head, but it is often the least clothed part. A simple beanie hat does wonders, with a scarf over the lower part of the face. If you tend toward black headwear like myself, the fact that you are riding a bicycle should allay any fears that you might be a terrorist.
Note: each of the clothing options described in this Instructable will be rated for cost, weather-resistance and looking stupid on a completely unscientific and wholly subjective 1-5 scale, with 5 being the preferred end of the scale (cheap, weatherproof and not stupid looking)
Weather resistance: *****
Looking stupid: *****
Weather resistance: **
Looking stupid: ****
Beanie hat and scarf:
Weather resistance: ***
Looking stupid: ***