You can substitute capacitors for chemical batteries, but the charge will decay if the feed is stopped. However, I have a dynamo powered flashlight that uses capacitors and works well.
Well, a lead-acid battery is simple and can hold a lot of charge, though usually not small and involving hazardous materials. (I don't know whether there's a recipe somewhere for gel cells -- do a websearch.) There's probably something equivalent which would be safer, though probably much builkier for the same power. I think you'll find that you can buy better and cheaper than you can build. But if you just want to do it as an experiment, go for it.
you can make lemon battery and try to recharge it, but that won't be very efficient