Sanding the inside of a hole can be horribly aggravating, especially on a pocket hole that doesn't go the whole way through the piece. You can try using a round file, but its hard to get consistent results, and a short range of motion really slows down progress.
In my case I needed to widen every hole in a handrail by about one millimeter so the spindles could be pushed further up inside. The holes were just slightly bigger than my pinky, so doing it by hand just wasn't an option, and I didn't want to use a drill bit, which would have splintered the edges of the hole.
The solution to this was to create a mandril that could be used in a hand drill, basically turning it into a hand-held spindle sander. I made mine at http://techshop.ws
Step 1: Choose an appropriate dowel to make your mandril out of
Each task is unique, so you get to tailor your tool to the task you want to use it for. I like for my mandril to be a little bigger than half the diameter of the hole I'm sanding. That makes it big enough that it won't get easily hung up in one spot and create a trough where it sanded away too much at once, but its still small enough that you can maneuver around inside the hole and give attention to where it's needed.