In my case I needed to widen every hole in a handrail by about one millimeter so the spindles could be pushed further up inside. The holes were just slightly bigger than my pinky, so doing it by hand just wasn't an option, and I didn't want to use a drill bit, which would have splintered the edges of the hole.
The solution to this was to create a mandril that could be used in a hand drill, basically turning it into a hand-held spindle sander. I made mine at http://techshop.ws
Step 1: Choose an Appropriate Dowel to Make Your Mandril Out Of
Step 2: Cut Your Dowel to an Appropriate Length
If the dowel you're using is too large to fit into the chuck of a drill, you'll need to decrease its diameter where it it will be clamped into the drill chuck. The concentricity of the two different diameters is very important if you want your tool to be balanced as it spins, so while you maybe could shave it down by hand, I doubt you'd get really good results. The obvious solution would be to use a lathe to remove the excess material.
Step 3: Cut a Slot for the Sandpaper
Use a coping saw or scroll saw with a rough toothed saw blade where the teeth extend slightly to the left and right of the blade. This will allow the blade to be able to cut the slight corkscrew we're trying to achieve.
Use a simple jig to keep the dowel centered on the blade as you make the cut by clamping piece of scrap wood to the saw table to push the dowel against to keep it stable.
Be sure not to twist the blade or you'll be likely to snap it. If you're having trouble cutting the twist, try alternating angling the blade slightly to the left and to the right as you progress, creating a slot that is wider than the width of the blade.
Step 4: Install the Sandpaper
The slot being at an angle can make it tricky to insert the sandpaper. I find it easiest to lay one end of the sandpaper over the slot and put a crease in it with my thumbnail, giving it the proper angle. It can then be folded and inserted into the slot. Wrap the sandpaper around the mandril and repeat the crease process for the second end of the strip. Trim off any excess sandpaper from the second end that won't fit down into the slot. If your slot is narrow with respect to the thickness of the sandpaper, it may be easier to (after creasing both ends of the strip) curl up the strip and slide it down over the mandril with tabs in the slot.
For additional security a rubber band can be wrapped around the sandpaper. Put the rubber band as close to the base of the mandril as possible to keep it out of the way.
Clamp the mandril in the drill chuck just like you would any drill bit. Make sure it is centered in the teeth as you clamp it down.
Keep the speeds low as you use your new sanding tool to help prevent the sand paper from being slung off of the mandril.