stop 7-segment from ghosting?

I'm working on a simple arduino 7 segment driving circuit. (It also has a led bar-graph that displays binary numbers but that part is working fine.) When I get the display working I might make an instructible out of it.

Here's the basic layout:
-using this 8 segmet 4 digit common cathode display.
-arduino controls what segments are on via 74HC164 8-bit Serial Shift Register
-shift register controls an array of 8 PNP transistors with base and pull-up resistors
-transistors connect to display anodes through resistor array
-display cathodes (common for each digit) connected to array of 4 PNP transistors with base and pull-down resistors 
-PNP transistors controlled by arduino.

(I will post the schematic shortly.)

In software, to display a four digit number, I shift the pattern for a given digit to the shift-register, which in turn turns on the PNPs for the corresponding segment anodes. I then turn on the NPN for that digit's cathode. I delay for a few milliseconds, then turn the NPN off, shift the next digit in, and turn the NPN for that digit on. Rinse and repeat.

It works fine but the segments that should be off are at about 1/2 brightness, making all the digits look like eights. Even the DP looks like it's on and I'm not turning it on anywhere.

Do I need pull-down resistors for the anodes or something? What's going on here?