Tell us about yourself!
On the surface, that seems like a nice idea. However, you'd lose half the choices you could make if you switched. Sometimes metric is cheaper, sometimes SAE is cheaper. If you are MAKING something however, the thread standard doesn't matter, because you can use what you have on hand.
That is the best way IMHO, since your accumulation of screws/nuts/bolts keeps getting added to as you go. The pull out thread gauges that allow you to count threads are a tool worth getting since it is so small you can tuck it away and pull it out when needed. Most tap and die sets come with these included. The problem with the setup here is that you could cross thread a part by putting the wrong size on, but the biggest weakness would be all the variations out there would make this board 6 feet tall. If you use an organizer for small parts it makes it much more compact and useful.