If you like that piece of retro hardware, but tired of replacing batteries every week - here's the solution!

Step 1: Extra Parts Needed

1) LiPo battery small enough to fit inside, I used 138mAh

2) SparkFun Power Cell - LiPo Charger/Booster

VMU works with minimum of 5V.
3.7v LiPo battery is enough to power VMU, but it'll give you "low power" warnings
so to get it to 5V you can just use a step-up or use charger/booster, I had one around so I went for this option.

Step 2: Free Some Space Inside

To fit everything I had to unscrew and remove metal contacts used for CR2032
Also to clear everything even more I snipped all extra plastic inside and made opening for USB connection.

Step 3: Put PCB and Battery Inside

Don't forger to solder jumper to 5v output
Solder output from Charger/Booster to battery input on VMU
Solder battery to input of Charger/Booster
Don't forget to isolate everything

Step 4: Put Everything Together

After checking that everything isolated and working put it back together and enjoy!
So now you can use your VMU as usual and just recharge a battery over USB.

If you want to take it further it's possible to use power from controller to recharge LiPo in VMU

Possibly I'll try to upgrade it a little bit more in future