The best solution would be to have a drill and a bit of the right size. But at that moment I didn't have one. Additionally, using a drill on the perfboard may result in complete destruction of the soldering pad, or maybe you don't want your drill to get too close to your circuit if it's already mounted.
Every time I have to solve a problem I think: "what would Mac Gyver do in my situation?". I just looked at my desk and I found the solution: a paperclip!
First, mark the holes you want to enlarge.
Then, straighten one side of the paperclip and insert it in the hole.
Turn it around as you were mixing your cappuccino.
Et voilà! The holes are safely enlarged and they can accomodate the oversize pins.