I'm writing this because I had this problem myself. Apple really doesn't give a straight forward way to downgrade. So after some searching I found out how easy it really is.
I'm going to keep this as short as possible. (not really... but I tried :) ).
Step 1: Backup Files You Wish to Keep
Back up everything you want to keep to an external harddrive.
(YOU WILL LOOSE ALL DATA CURRENTLY ON YOUR COMPUTER!)
Step 2: Internet Recovery Mode
(First, turn your sound up. Trust me.)
Restart your Mac and get ready to press and hold "Command" "Option" and "R".
If you only hold "Command" and "R" you will end up re-installing Yosemite. Be sure to hold all 3.
When you hear the Mac start up tune press and hold all 3 keys. This will send you into "Internet Recovery"
(It will ask you to enter your wifi password. Mine wouldn't work and I had to use a LAN connection. You may or may not need to do the same).
Step 3: Erase Your Harddrive.
Select "Disk Ultility" and press continue.
Select your Main Harddrive (where OSX Yosemite is installed)
Select the "Erase" tab.
Make sure its is set to "Mac OS Extended (Journaled)".
Name the disk something like "MacHD" Click "Erase"
NOTE: After this step all of the data on your harddrive will be gone forever.
Step 4: Install Your Original OS
Exit the "Disk Utility"
Select "Re-install MacOSX ". (This will install whatever version of OSX came with your mac when it was first purchased. For me it was Lion. If your original OSX was Mavericks then you are done. If not.... Continue)
Step 5: Upgrade to Mavericks
Now you have Lion installed. (or other OSX before Mavericks)
Go to the "App Store" app and select "Purchases" (don't forget to sign in).
The first item will probably be Yosemite. DO NOT CLICK IT!
Scroll down and you will find Mavericks. Press Download and wait and unreasonable amount of time.
This worked for me. I hope it works for you.