Wash it! Really, you should wash all of your material. If you're using white and black then you might not want to wash them together, because of all the lint and fluff that will come off the black and stick to the white. After I dried my grey fleece I wiped down the inside of our dryer becuase of all the lint. It wouldn't make much sense to wash them separately and then put one color of fleece in the dryer only to get fluff of the wrong color all over it.
After the felt was washed and dried it looked like a clown had exploded in our dryer.
I washed all of my material on a warm/cool setting and then dried them on a normal/less dry setting.