A podcast is just an mp3 file. It can hold audio content, like music or talking.
A podcast with no content is sort of boring.
So you have to figure out what your podcast will be. Will you start your own radio talk show? Will you share your musical compositions with the world? What is your goal with this podcast? Before you do anything else, you need to figure this out.
Step 1: Recordificate
Once you've figured out what you want to be in your podcast, the next step is to record it. You don't need lots of fancy software and hardware for this. You simply need a microphone of some sort and some recording software (same goes for recording yourself playing music).
You can get a microphone for your computer very cheaply almost anywhere. I got one from a yard sale for 10 cents. Just plug it in and fire up your recording software.
I also have a sandisk sansa e250, which comes with a voice recording feature. It records to .wav files, which are easily convertible to .mp3. You could even record yourself using an old cassette recorder and then rip it to .mp3 if you like.
For the software, you can use the free audio software audacity. Windows also comes with the stupidest sound recorder ever which nonetheless will work for this if you insist on using it. If using the non-vista version of windows sound recorder, just record some audio and copy/paste it until you have an audio file of the length you want to record for. Then record over it.
So get a microphone, get some audio software and record your content! Then you can publish it.