Actually, we can attach a vacuum holder to quite a rough surface with the help of reusable adhesive material called Patafix.
In case you are not familiar with it it's very useful stuff with a texture of an (already chewed) gum. It easily sticks and un-sticks just like Post-It notes and it also can be bought in Office Depot.
After you press down the holder the material will fill the small bumps on the surface and hermetically seal the cup.
Note that the Patafix was applied not on the extreme outer edge of the cup but rather a little bit closer to the center. This is because there is a plastic cup above the silicone vacuum cup and we need to have the Patafix under (on this upside-down picture it's above) the rim of the plastic cup.
Thus when you press the plastic cup it will apply greater force to the layer of the Patafix.