Power Up ANYTHING With Old Cell Phone Chargers.

Actually you should not use the pot at all. It is inefficient, cannot supply much current, and does not regulate. Use a regulator or just stick to rated voltage. A pot should generally be used only for a fixed, know load, and after the revelant equations for power and voltage are checked.

You might find a 50ohm rather than 50kohm pot more effective and give wider control stability but as you stated a linear regulator would be far superior