A few months ago I did some experiments with gussying up my keyboard using white keys and RIT Dye.
Initially the boards looked very spiffy - bright and uniform.

However, after a mere week of daily typing, the keys started to fade. It wasn't just from usage either; I found untouched keys also losing their colors [perhaps from sunlight].

A myriad of sealing-technique attempts later, I found nothing would preserve the color to my liking.

Take this as a cautionary tale- I've seen a few threads now of people asking for advice regarding the longevity of dyed keys. Sadly, it is not great. I wouldn't recommend this technique at all. It's definitely worth the money to invest in a nice mechanical keyboard and buy the appropriate caps in the colors of your choosing.

Step 1: Keycap Resources

For the uninitiated: once you've got your mechanical keyboard, you'll want to check what sort of switches are under the keys so you purchase caps that fit. There are slight variations in dimensions as well as shape of the stem.

The first stop for a lot of folks is eBay: the caps sold by most merchants are cheap - both in price and quality. This isn't to say they'll break or fade, rather that the overall feel of the keys is 'thinner' and the lettering will likely be painted/stuck on. But if you're just in the mood for a little fanciness on your board, or if you need a huge variety of caps for some insane color scheme, there's nothing wrong with these keys.

Reputable Sellers:
Elite Keyboards
WASD
E-Sports Gaming

And of course, your most important resource should be http://geekhack.org/.
This is the forum for all things keyboard related. In respect to key caps - they occasionally do group buys, so you can put in your order and not have to pay an arm and a leg for having custom caps manufactured.