I have a 'thing' for magnets; I have dozens and dozens of them and all of them are working for me - doing something useful.
Step 1: Easy to Find / Handy to Use
This is how to do it:
First, clamp the metal frame of the speaker in a bench vise. Clamp it good and tight.
Second, using a dull chisel, separate the magnet from the frame. They are usually riveted together, except the smaller, cheaper models. Just position the end of the chisel where the magnet and frame join and whack it soundly with a hammer.
The magnet will fall off, usually with the first hit.