Tell us about yourself!
I was confused about why the LED for pin 13 stays lit too. As it turns out, the LED is not driven directly by the pin - there is a buffer / driver that monitors the voltage on the pin. Pin 13 is configured as an input if not used, and so the voltage on the pin floats high enough for the buffer to turn on the LED. Connect a 1k (or whatever you have) resistor from ground to pin 13, and the LED goes out.