The particular challenge was limiting the code size in order to fit into the 2 kB of memory in the Attiny2313. It worked by storing the character font matrix in the program memory space PROGMEM and by using direct port manipulation in combination with bitwise math operations.
The pin mapping with the Kingbright TC07-11EWA 5x7 LED matrix is:
Attiny2313 port pin LED matrix pin (470 Ohm resistor)
PB0 12 6 Row 7
PB1 13 5 Row 6
PB2 14 4 Row 5
PB3 15 9 Row 4
PB4 16 2 Row 3
PB5 17 11 Row 2
PB6 18 12 Row 1
PD0 2 1 Column 1
PD1 3 3 Column 2
PD2 6 10 Column 3
PD3 7 7 Column 4
PD4 8 8 Column 5
PB5 (pin 9) is the input pin for the button. Pressing the button will connect the button pin to GND.
The power supply is via a 7805 linear regulator with two 100 nF capacitors for oscillation suppression. On the photo they are hard to see. I also added a socket and a power switch. Mechanical stabilization was realized with the indescriminate use of a hot glue gun.
It seems that directly driving an LED matrix with a microcontroller is not the orthodox approach but the Attiny2313 hasn't complained yet.
Had I given more thought to the circuit board layout I would have left more space between the Attiny2313 and the LED matrix to accomodate another signal line. This line is now around the matrix and then back to reach pin 12. With a little more planning I also would have placed the 100 nF capacitors somewhere else so I could have bent the 7805 regulator downwards. Then everything might actually have fit into the old music cassette case.
I ignored a decoupling capacitor between VCC and GND of the Attiny2313 because I figured that the capacitors at the 7805 regulator are good enough. There is also no debouncing of the input button because there is a delay included in the software. The simplicity of the circuit did not warrant a reset button. Finally, I did not place a 10k resistor between pin 1 (RESET) and VCC as suggested in some minimal circuits because other sources mentioned that there was an internal pull-up resistor anyway.
As no crystal is present, the microcontroller uses its internal RC oscillator. I found the timing to be a bit off so I had to empirically determine delay values in the software. After accidentally loading a bootloader onto the Attiny2313, the timing seemed to get better. I have no explanation for that.
In order to program the Attiny2313 I used my Arduino as an ISP with the appropriately named "ArduinoISP" sketch. The directions in the sketch are straightforward. In preparation, I followed these instructions using this source for the required core files.
As I am using an Arduino Duemilanove, I had to place a 120 Ohm resistor between the RESET pin and VCC to prevent error messages during programming. This is described here.
I hope the attached code is self-explanatory.