There are other, more-elegant (but more-expensive) solutions, of course. For instance, you could take your hymnal to a bookbinder and have it spiral-bound. Churches have also been known to photocopy dozens and dozens of hymns (because, ultimately, loose-leaf is the best format for print music), but this is frowned upon in some circles because it's illegal.
I love old church music and I recommend an exploration of it to any keyboard or voice student studying music from the Baroque period to the 1950s, give or take. I also recommend that any Christian be somewhat familiar with the old music, because it discusses a surprisingly different range of topics than the music we sing today.
Jesus loves you whether you believe it or not. Happy musicking!