This shrine you're likely to discover only in your travels. No one mentions the site on the southeast bank of the Silverfish River east of the Imperial Bridge Inn---indeed, no one talks about the god Peryite at all---and its not close to any of Cyrodiil's cities.
Step 1: How to Level Your Combat Skills Faster in Oblivion: the Elder Scrolls IV
Strange place. The five worshippers, arranged in a circle west of the shrine, are all frozen in place. Try to activate one to learn he or she is unconscious. The followers tried to summon the god with a spell. It evidently backfired or a resentful god fired back, and now the bodies of the five rest in the mortal plane while their souls reside in oblivion. To use this method do not activate the shrine! Since these npc's are considered necessary to fulfill this deadric quest they cannot die, luckily the game does count them as target.