Ok, you've got everything wired up, you know you've got power, but the question is, did you wire everything up correctly? Let's check.
For this you'll need a resistor, and LED and some code.
Wire up a resistor and an LED as shown in the photos. For this one, I used a 330 OHM resistor, and a red LED. Take note of how you plug in the LED - they have polarity - the short leg goes into the negative/ground rail, and the longer, positive lead goes to the ATMega chip...pin 19. As before, if you're not sure what pin this is, refer to the Arduino mapping image in Step 4.
Now, you need to download the Arduino Sketch I've attached, open it in the Arduino software, and upload it to your chip. This will make Arduino pin 13 (but it's ATMega pin 19 as I mentioned in the previous paragraph) blink every second. It's from this great Getting Started with Arduino book I have.
Once you've hooked up your LED and resistor, programmed your chip, put it back on your breadboard, then you can connect up your power. You should get a blinking LED, which means you've hooked up everything correctly!
Below is a brief video of the circuit we just built with this blinking LED: