Make the bearing a better fit. Not too loose.
There are no bearings in a pinewood derby car. Official cub scout rules only allow the plastic wheels and nails that come with the kit. No bearings or axles allowed in an official race.
In this case the hole through the wheel is effectively a bearing surface - and, if too big will induce chatter.
Break it in. Use the graphite lube and work it into the wheels. If you have a treadmill then run the car on it for a while to break in the wheels with the nails.