Sanding the inside of a hole can be horribly aggravating, especially on a pocket hole that doesn't go the whole way through the piece. You can try using a round file, but its hard to get consistent results, and a short range of motion really slows down progress.
In my case I needed to widen every hole in a handrail by about one millimeter so the spindles could be pushed further up inside. The holes were just slightly bigger than my pinky, so doing it by hand just wasn't an option, and I didn't want to use a drill bit, which would have splintered the edges of the hole.
The solution to this was to create a mandril that could be used in a hand drill, basically turning it into a hand-held spindle sander. I made mine at http://techshop.ws
Step 1: Choose an Appropriate Dowel to Make Your Mandril Out Of
Each task is unique, so you get to tailor your tool to the task you want to use it for. I like for my mandril to be a little bigger than half the diameter of the hole I'm sanding. That makes it big enough that it won't get easily hung up in one spot and create a trough where it sanded away too much at once, but its still small enough that you can maneuver around inside the hole and give attention to where it's needed.
Step 2: Cut Your Dowel to an Appropriate Length
Again, you're making a custom tool for a specific job, so the length of your mandril will be determined by how deep you need to get sandpaper into the hole. Take that length, add a half inch or more that will act as a spacer between the sandpaper and the drill, giving you some maneuvering room while sanding, then add another inch or more that you'll clamp into your drill chuck.
If the dowel you're using is too large to fit into the chuck of a drill, you'll need to decrease its diameter where it it will be clamped into the drill chuck. The concentricity of the two different diameters is very important if you want your tool to be balanced as it spins, so while you maybe could shave it down by hand, I doubt you'd get really good results. The obvious solution would be to use a lathe to remove the excess material.
Step 3: Cut a Slot for the Sandpaper
The slot should have a slight twist to it as it travels up the shaft. This will help push the sandpaper towards the base of the mandril where it clamps into the drill instead of getting spun off the tip. That assumes, of course, that the drill is spinning in the intended direction.
Use a coping saw or scroll saw with a rough toothed saw blade where the teeth extend slightly to the left and right of the blade. This will allow the blade to be able to cut the slight corkscrew we're trying to achieve.
Use a simple jig to keep the dowel centered on the blade as you make the cut by clamping piece of scrap wood to the saw table to push the dowel against to keep it stable.
Be sure not to twist the blade or you'll be likely to snap it. If you're having trouble cutting the twist, try alternating angling the blade slightly to the left and to the right as you progress, creating a slot that is wider than the width of the blade.
Step 4: Install the Sandpaper
Cut a strip of sandpaper whose width is the length of the slot you cut in the mandril. I prefer the fold and tear method instead of dulling a knife by actually cutting sandpaper, but you can mangle your own tools however you see fit.
The slot being at an angle can make it tricky to insert the sandpaper. I find it easiest to lay one end of the sandpaper over the slot and put a crease in it with my thumbnail, giving it the proper angle. It can then be folded and inserted into the slot. Wrap the sandpaper around the mandril and repeat the crease process for the second end of the strip. Trim off any excess sandpaper from the second end that won't fit down into the slot. If your slot is narrow with respect to the thickness of the sandpaper, it may be easier to (after creasing both ends of the strip) curl up the strip and slide it down over the mandril with tabs in the slot.
For additional security a rubber band can be wrapped around the sandpaper. Put the rubber band as close to the base of the mandril as possible to keep it out of the way.
Clamp the mandril in the drill chuck just like you would any drill bit. Make sure it is centered in the teeth as you clamp it down.
Keep the speeds low as you use your new sanding tool to help prevent the sand paper from being slung off of the mandril.