UPDATE 04/04/2016 : Multipoint software + Emulating NES video/pictures + OPTION_7 Point and Click + speed test + OPTION video
UPDATE 03/24/2016 : CCD sensor reverse Engineering + Using 2 CCD sensors + Laser engraving vector files + Retropie playing
Hi and welcome to my third Instructable !
I have been reading about DIY touch screen table projects for a while now, and I find most of them very complicated. Often They imply using a videoprojector, reflection mirror and IR camera. The result gives you a complete multi touch screen solution, but you need some depth for the videoprojector to have a big screen. Not the best solution for a coffee table...
I made this tutorial to help people to build their own touch screen solution based on any flat screen TV. It uses light triangulation technology to read the coordinates. This method is quite cheap ( the priciest part is the TV), with a good quality image and you can make the table almost as small and as thin as your TV is.
Of course, nothing is perfect and with this solution it's not possible to build a 100% multi-touch screen and it's hard to cover the entire screen area. But depending on the way you make it you can have some interesting results. Plus, I am sure there are a lot of possibilities I haven't thought about to improve this.
This solution implies the use of light triangulation to read a very precise coordinate. Basically, it's like having a huge number of IR receptors with some IR LED. You turn on every LED one by one and read the sensors. When one receptor can't see any light, it means that an object is blocking the light. With enough LED and sensors you can get a much more precise solution.
But for this project, I was looking for something more suited for the reading. A CCD sensor, is in every scanner. In those devices, it's basically a black and white camera which reads only one line of pixels. To scan a color document, a RGB LED blink in the three colors and the CCD reads all of them for every line of the document. Then it can calculate the exact colors depending of the Red Green and Blue reaction.
Using a CCD sensor gives you access to 2700 light sensors on a 20cm line. Most people who take sensors and actuators from an old printer/scanners only take the RGB LED and the light conductor from this part, as we can see here. If you like recycling, you could examine the possibilities of CCD. Not having found many DIY projects on the internet using them, there are probably a lot of different ways with which you could use these sensors! In this tutorial I will also explain how to hack your own scanner.
For this touch table, I used one CCD sensor with 24 IR LEDs. The object is detected on a 740x380mm surface by a Teensy board (which works under Arduino environnement). This board turns the IR LED ON, reads the CCD and calculates the exact position of an object. Then it sends the coordinates as a mouse, a keyboard, a Python program or just as Serial on a Raspberry Pi which shows the interface through the HDMI on a 32" TV screen.
When I am working on a project, I like to build things that are the most cost efficient possible but still of good quality. So even those on a tight budget can afford to build their own. It's hard to give the exact amount of money spent, as many things are second hand or from the tip. Here is an approximation :
Some parts can be a bit difficult to solder, I recommend having a good soldering iron. At home I have been using a WS81 from Weller. A bit expensive to buy but it's precise and you can keep it for 20 years ;-)
The wooden box and table :
This part is gonna depend on your TV screen. This is for my 32" LG TV :
The TV box :
The Table under :
Approximate cost of the wood 70$
I also used a Trotec laser cutting machine to engrave some characters on it and to cut several small parts of the wood. You can rent it for 25$ an hour from the Fablab next to my place. As I often help them on electronic projects I can sometime have it for free :-)
Touch detection :
As Wikipedia is way better than me on explaining :
"A charge-coupled device (CCD) is a device for the movement of electrical charge, usually from within the device to an area where the charge can be manipulated, for example conversion into a digital value. This is achieved by "shifting" the signals between stages within the device one at a time. CCDs move charge between capacitive bins in the device, with the shift allowing for the transfer of charge between bins.
The CCD is a major piece of technology in digital imaging. In a CCD image sensor, pixels are represented by p-doped MOS capacitors. These capacitors are biased above the threshold for inversion when image acquisition begins, allowing the conversion of incoming photons into electron charges at the semiconductor-oxide interface; the CCD is then used to read out these charges. Although CCDs are not the only technology to allow for light detection, CCD image sensors are widely used in professional, medical, and scientific applications where high-quality image data is required. In applications with less exacting quality demands, such as consumer and professional digital cameras, active pixel sensors (CMOS) are generally used; the large quality advantage CCDs enjoyed early on has narrowed over time."
Basically, the CCD you find in a scanner doesn't read lights in two dimensions, but on one line. With the one I tried, there are 2700 different light detectors that we can use as 2700 pixels on a X position.
As I couldn't find any tutorials explaining how they work on the internet, I tried to understand the logic by myself. I started by taking a printer scanner from the trash and opening it. Please be very careful with these devices, they use 220V! Don't do it if you are not sure of what you are doing. Always manipulate them when it's not plugged in and isolate any high power part.
By visually checking all the lines going from the control board to the CCD you can try to identify the 4 lines controlling the RGB LED, they are generally located at the extremity of the CCD board. With a multimeter, you can check where those lines are going on the communication flex.
Then you can check if some lines are bigger than others (it's probably going to be the power and the ground, you can verify which one it is with your oscilloscope when the power is ON). Plug an oscilloscope between the other lines and the ground to understand what they are doing.
I tried two CCD sensors from different brands and they where both using the same communication protocol but not on the same "line order on the flex". When I turned the scanner ON, I could see some data passing. It means that the device was probably testing the sensor when it is turned ON.
- One line was giving a perfect square signal, I deduced that it was a signal clock.
- One signal was turning ON and OFF at the beginning of the clock transmission, it's the Latch signal
- one signal was sending a lot of analog value. Those values changed when my hand was next to the sensor. It is the values from the CCD.
So I made a test program the on a Arduino UNO board. You can download it at the end of the step
With this, you can read on the serial port 16 average values (I call them pixels) of 168 light sensor on the sensor, so almost the 2700 total. Just plug in the lines as explained at the beginning of the above file on any Arduino board and read the data on the serial monitor.
Last week, I was trying to find some solutions to improve the cover of the detection area. To do this, I wanted to have a second sensor to test new configurations and speed. So I went back to the tip and I took several printer/scanner from different brands.
I was a bit frustrating to discover that some scanner have a different kind of CCD sensor. No need to open the entire device, they are pretty easy to recognize, as the black plastic package is way bigger (picture 1). Inside, they have a way more complex mirror architecture, a lens and... a very tiny CCD sensor! I will try to hack those one to see the possibilities soon, but for now, I needed another "normal" 22cm CCD. So I moved on!
Second printer I opened was from EP*** brand (Picture 2). The flex connection had 12 lines on 5V :
After few hours of test, I couldn't find the good protocol to have the OUTPUT response. Even when the CCD was plugged in the scanner, the analog wasn't giving any clear response (on startup test as the printer was then reporting an error).
So I moved on to a third printer from H* brand. The flex had 14 lines on 3.3V :
I could access some PAD from the different line (picture 3). After a bit of testing I succeed to have a clean answer on the analog line (pictures and explanations 4 and 5).
The CCD I was using since the beginning is from CA*** brand. Here are the lines :
I had now to build a little board to be able to control the two CCD with the same latch and clock signal.
The connection for this board are :
So I send the Latch signal, then I read the two Analog at every clock period. Most of the pictures and video have only one sensor, it's because I made them before upgrading the table :s
Now we have a working CCD sensor with Arduino, we can start building our first touch screen:
You can take the glass from the scanner to use it as the surface. On one side you add the sensor and on the other side a IR LED. Power the LED (don't forget to add a resistor depending on the spec!) and start reading the sensor, you will see that a part of the serial data will raise. On a 10 bit ADC like the one on Arduino, the range value is 0 to 2^10=1024. On my test, my value is always between 550 to 980.
Two important things :
To visualize this I made a little processing program that you can download at the bottom of this step. This program shows 20 different cluster of the CCD sensor. If the average analog value of the zone is close to 1024, the cluster is white. If it is lower it gets darker. As Processing needs to communicate with "Serial.write" data and not "Serial.print" like the Arduino Serial port, you need to upload this new Arduino code on your board.
Now, if you put your finger between the LED and the sensor, you should see the cluster getting darker. Congratulations! You have made your first touch interface!! Now you can detect the area where your finger is, not very precise, but functional!
To improve the precision of the detection we are going to have to complexify the system by adding some LED on different digital ports of the board. As you can see on the pictures, I now have 15 LEDs (choose as many as you want) wired in a plastic brown sticks. The distance between each LED is 5cm so the total distance is 70cm.
Now, how are we going to detect a very precise position?
1. we need to check if the system is detecting something. The method is gonna be :
if something is in the way store the pixels above calibration in a table
Now we have stored all the areas detecting something.
2. calculate the function
I know school is often far away and mathematics didn't seem very useful at the time. But do you remember the basic function Y = AX + B ?
Here we are going to have a full application of this. We are going to calculate the function crossing the LED and the middle of the area the CCD sensor detected.
To simplify, we are going to set the XY origin of the graphic on the first LED called LED 1. We can then calculate B for every LED (in mm) :
LED 1 = 0, LED 2 = 50 (50mm distance), LED 3 = 100, LED 4 = 150...
The A is a bit more tricky, if Y = AX + B then A = (Y - B)/X
If we take the average of the pixels detected (in the program it's "(maxi + mini)/2") we can calculate the exact height (Y). We also need to transform the pixels in dimension.
and as we have Y, B and X we calculate A :
A = (Y - B)/X
Now we have A and B of all the lines going from every LED to the middle of the area they can't see on the sensor. We have function1 : Y = A1 X + B1 and Y = A2 X + B2
We can then calculate the crossing of those functions :
The coordinate X and Y of the crossing will be :
X = (B2 - B1) / (A1 - A2)
Y = (((A1*X) + B1) + ((A2*X) + B2)) / 2
The X and Y are the crossing coordinates. If you calculate all the functions crossing point and you average them you should have a good approximation of the central position of the object.The multi-point can't work with only averaging, it need a more complex algorithm (in step 9).
As we can see on the picture, this method is not really perfect. With 15 LEDs on one side, you can see on picture 1 there are still many areas we cannot detect.
So how are we going to make the detection area bigger?
Using 2 CCD sensors
My new solution was to add a second CCD sensor (picture 7). Now, the entire screen is almost covered! But as two sensors is slowing down the system, I can play with the number of LED I use, and with the number of pixels. This way I am quiet flexible on the cover, the precision and the speed I want!
For example, if I need to emulate a keyboard for a Nintendo Nes, I only need 8 buttons. I can split my screen in 8 parts and detect if something is on this zone. Only 30 pixels (15 par sensor) and 8 LEDs is way enough to do that and I can reach around 40Hz speed.
If I want to emulate a mouse I need access to the entire screen with some precision. So I ll use 100 pixels (or more) and 16 LEDs. The speed will drop down to 15Hz but I can click where I want!
On the video, I show the Processing program running on the Raspberry Pi. I only used one CCD as it was before the 2 CCD upgrade, but the position still takes some time to synchronize. This problem is only because Processing is a heavy program, so the Raspberry Pi is struggling to refresh and synchronize the screen and the serial port fast. I had to add a delay on the Teensy program to slow down the communication. If you try the Processing GUI on a more powerful computer, it will be perfectly synchronize.
As my LEDs don't have a big enough angle to cover the two sensors, I added a second LEDs in parallel pointing to another direction. I just had to adapt the resistor to have twice the power, and it's done!
As I was saying, the multi-point cannot be perfect using a light triangulation technology with a CCD sensor. It is pretty complex to work on this part... But it is also very fun to imagine many different solutions on how to have the best results without adding problems.
Here is how I made mine :
This algorithm isn't perfect, but it allow me to use some multi-point for games, as I can choose where the detection points are (see the NES remote on step "Final program, let's play!"). I will keep improving it, and hope some people will advice and help me as well :-)
On the Fritzing picture, you can see how everything is connected.
A good thing about using a Teensy Board is you have 34 digital Input/Output. So we have enough free digital PIN to control 30 LEDs without the need of external LED drivers. As only one LED is on at a time, we don't need much power either.
In this project we are going to use the following Pin :
I now have two LEDs instead of one for every PIN (to improve the angle for two sensors). The LEDs are powered on their Anode, which means you have to set your Pin Low to turn the LED ON. A 110ohm resistor is wired on the power line to avoid sending to much power on the LEDs.
I have already talked about the great hardware specifications of the Teensy board. But combined with the Arduino software environment, the Teensy also offers some great possibilities. But first let's install the plug-in
Now you should see a category "Teensy" when you choose your board. And if you select it, you have access to several other options like the "USB Serial Type" to instantly transform your board into a USB keyboard or mouse (some possibilities can be found on Arduino Leonardo). The USB communication type is a "Full Speed" which means it can communicate at 12Mb/s. You can also change the speed of the board (on this project I used the overclock at 92Mhz).
For now, all of my Arduino UNO codes have been successfully built on the Teensy which means the adaptation between the ARM Cortex M4 and the AVR Atmega328 have been very well made. I tried to write the code to be understandable. Here is the structure of the main loop :
- Turn an LED on (as the LEDs are wired with the Anode on VCC and Cathode on the PIN, you have to give a LOW state to turn it ON)
- ClearCCD(), The sensor has to be cleared once before reading the value... I don't know why. If anyone has an idea of how I could speed up this part, I would very much appreciate it.
- readCCD(), read the sensor, if a value is above the calibration it is stored and the function return 1. If something have been detected, the flag_detect is set to 1.
- Turn the LED off (HIGH state for OFF in this configuration)
- OPTION 1; send Serial monitor.
Basic debugging : This option allows you to read the 16 clusters directly on the serial monitor.
- OPTION 2; calculate position + send Serial monitor
Debugging position : send the coordinates X and Y to the serial monitor
- OPTION 3; send Processing interface
Debugging visual : look at the CCD results on a Processing GUI. The interface is going to have to be adapted to the number of LED (16 default).
- OPTION 4; calculate position + send Python Pi interface
Send the position to the PyQT interface
- OPTION 5; calculate position + send mouse (only on Teensy)
Send the position as a USB mouse interface. To use this option you will have to go to "Tools" -> "USB Type" -> "Keyboard + Mouse + Joystick". (See the Warning at the bottom of this Step)
- OPTION 6; calculate position + send Keyboard (only on Teensy)
Send the position as a USB keyboard interface. To use this option you will have to go to "Tools" -> "USB Type" -> "Keyboard + Mouse + Joystick".
- OPTION 7; Point and click. Same as mouse but click instead of following the position
WARNING: OPTION 5 uses the command mouse.mouveTo(X, Y) which doesn't work on a Linux station. If you want to do the same configuration as me with a Linux Raspberry Pi you are going to need to use the UGLY HACK made by Paul Stoffregen who seems to be one of the developers of the Teensy Project. He is really good and he helps a lot with debugging if you have any questions. Thank you Paul!
For this hack, you need to modify a part of the file name usb_desc.c It should be installed in your Arduino folder. The new configuration given by Paul is slightly different but he makes it work! By then, the mouse.move() function is not working anymore.
On the Raspberry Pi side, there are many different ways to show some data with a GUI. I choose to write a basic interface with PyQT following a friend's advice. Now, the interface has a full screen picture, read the serial datas and write them on a little panel on the left of the screen and draws a cursor following the serial position received.
The program run on Python3.2 and Pyqt4 on the Raspberry Pi. You can easily find many tutorial to install them and run a program.
As Raspberry Pi had made a good release on installing and running Processing on the board. I also decided to adapt my debugging code to show it on the big screen, it's very slow but but great for debugging and improving the Teensy code. To do this, you will have to rotate the screen 90° (If your sensors and LEDs are on both sides like me).
Now your screen is rotated, you can play with the number to have different rotation (2 is 180°, 3 is 270°, 0x10000 for horizontal flipn 0x20000 for vertical flip)
Another tricky part was building the detection area. I didn't know which material to choose so I started with glass panels. But it was too fragile to manipulate and cut so I moved to plexiglass.
When everything is fixed and working. You can add some transparent glue above the surrounding plexiglass lines and around the LED to create an isolation if any liquid falls on it.
The wooden enclosure depends exclusively on the TV you have. I will only give the basic specification for mine, but I think the pictures are going to explain it better.
The little table under is made with :
In this step, I added some of the illustrator files I used for the engraving.
For the others, they are from this awesome website. It's a really good database of Adobe Illustrator Vector files of games (like Zelda, Mario, Mortal Combat or Street Fighter...), movies (Star Wars, Despicable Me), Comics (Superman, Batman, Deadpool...) and many other categories!
Now we have everything fitting together, let's try a few games :)
With the keyboard option of the Teensy program you can emulate any key based game. I made few test with the PyGame already on the Debian Pi like the game squirrel.py. Then, I installed Retropie on a second SD card and tried playing with few games.As the engraving of the table, I have the games Mario, Zelda, Star Wars and Pac-Man (and others!)
But my keyboard code wasn't adapted to emulate a gaming keyboard so I had to make few modifications on pressing and holding the keys. If you use the normal program, the frequency of sending the key is not high enough, so Mario is gonna move very slow, like if you where just pressing the button very quickly instead of holding it.
My new keyboard game code hold the position of the button while it's reading the next one. With this configuration the movement of Mario are smooth :-) . I also implemented the multi-point as you need to be able to move and jump in the same time.
As we can easily access the usb port, it's also possible to plug a controller in and to emulate some games without a touch screen.
Still a lot of thing to do, it's hard to work on everything in the same time :-)
I hoped you all enjoyed this Instructable and that it will inspire many other projects using CCD sensors or coffee tables!
As you can see in the last step, there are still many ways to improve it... If you have any ideas or any questions, I would be happy to work with other people on making it more reliable, efficient and fun! ;-)