Introduction: Virtual Touch Screen Game Using Zybo
Virtual Touch-screen Game hands-on tutorial for the Zybo will provide step-by-step instructions for customizing your hardware to emulate touch screen on simple TFT monitor using camera and finger detection.
Required Hardware:
- Zybo Board
- Vivado 2014.1 Webpack
- USB web camera
- VGA TFT monitor
- two Digilent JoyStick modules
- micro SD card (4GB or more)
- keyboard
- mouse and
-USB HUB.
Required software:
- Xillinux by Xilybus
- OpenCV
Step 1: Installin an Operating System and Configuring Hardware
The first step is to install an operating system on your Zybo board. The system that we chose is the Xillinux by Xillybus which is basically a modified Ubuntu 12.04 distribution of Linux based Operating System optimised for Zybo board.
First you need to dowmload two files:
- boot partition kit for your board - http://xillybus.com/downloads/xillinux-eval-zybo-...
- SD card image - http://xillybus.com/downloads/xillinux-1.3.img.gz
The instructions for what to do with these files are on the following link:
http://xillybus.com/downloads/doc/xillybus_getting...
After booting up the system, to run the GUI, you have to type in the command "startx" (without " ") and press enter (Picture 1). Now the GUI will load and you can continue with the project (Picture 2).
Step 2: Installing OpenCV
OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. OpenCV was built to provide a common infrastructure for computer vision applications.
To install and configure OpenCV 2.4.1, complete the following steps.
The commands shown in each step can be copy and pasted directly into a Linux command line. To open the terminal, press alt+ctrl+t.
1. Remove any installed versions of ffmpeg and x264.
sudo apt-get remove ffmpeg x264 libx264-dev
2. Get all the dependencies for x264 and ffmpeg.
sudo apt-get update
sudo apt-get install build-essential checkinstall git cmake libfaac-dev libjack-jackd2-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libsdl1.2-dev libtheora-dev libva-dev libvdpau-dev libvorbis-dev libx11-dev libxfixes-dev libxvidcore-dev texi2html yasm zlib1g-dev
3. Download and install gstreamer.
sudo apt-get install libgstreamer0.10-0 libgstreamer0.10-dev gstreamer0.10-tools gstreamer0.10-plugins-base libgstreamer-plugins-base0.10-dev gstreamer0.10-plugins-good gstreamer0.10-plugins-ugly gstreamer0.10-plugins-bad gstreamer0.10-ffmpeg
4. Download and install gtk.
sudo apt-get install libgtk2.0-0 libgtk2.0-dev
5. Download and install libjpeg.
sudo apt-get install libjpeg8 libjpeg8-dev
6. Create a directory to hold source code.
cd ~
mkdir src
7. Download and install install x264.
cd ~/src
wget ftp://ftp.videolan.org/pub/videolan/x264/snapshot...
tar xvf x264-snapshot-20120528-2245-stable.tar.bz2
cd x264-snapshot-20120528-2245-stable
8. Configure and build the x264 libraries.
./configure --enable-shared --enable-pic
make
sudo make install
9. Download ffmpeg version 0.11.1 from http://ffmpeg.org/download.html.
cd ~/src
wget http://ffmpeg.org/releases/ffmpeg-0.11.1.tar.bz2
tar xvf ffmpeg-0.11.1.tar.bz2
cd ffmpeg-0.11.1
10. Configure and build ffmpeg.
./configure --enable-gpl
--enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-nonfree --enable-postproc --enable-version3 --enable-x11grab --enable-shared --enable-pic
make
sudo make install
11. Download and install install a recent version of v4l (video for linux) from http://www.linuxtv.org/downloads/v4l-utils/. For this guide I used version 0.8.8.
cd ~/src
wget http://www.linuxtv.org/downloads/v4l-utils/v4l-ut...
tar xvf v4l-utils-0.8.8.tar.bz2
cd v4l-utils-0.8.8
make
sudo make install
12. Download and install install OpenCV 2.4.2. Download OpenCV version 2.4.2 from http://sourceforge.net/projects/opencvlibrary/fil...
cd ~/src
wget http://downloads.sourceforge.net/project/opencvli...
tar xvf OpenCV-2.4.2.tar.bz2
13. Create a new build directory and run cmake:
cd OpenCV-2.4.2/
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE ..
14. Verify that the output of cmake includes the following text: found gstreamer-base-0.10GTK+ 2.x: YESFFMPEG: YESGStreamer: YESV4L/V4L2: Using libv4l Build and install OpenCV.
make
sudo make install
15. Configure Linux. Tell linux where the shared libraries for OpenCV are located by entering the following shell command:
export LD_LIBRARY_PATH=/usr/local/lib
16. Add the command to your .bashrc file so that you don’t have to enter every time your start a new terminal.
Alternatively, you can configure the system wide library search path. Using your favorite editor, add a single line containing the text /usr/local/lib to the end of a file named /etc/ld.so.conf.d/opencv.conf. In the standard Ubuntu install, the opencv.conf file does not exist; you need to create it. Using vi, for example, enter the following commands:
sudo vi /etc/ld.so.conf.d/opencv.conf
G
o
/usr/local/lib
:wq!
17. After editing the opencv.conf file, enter the following command:
sudo ldconfig /etc/ld.so.conf
18. Using your favorite editor, add the following two lines to the end of /etc/bash.bashrc:
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig
export PKG_CONFIG_PATH
After completing the previous steps, your system should be ready to compile code that uses the OpenCV libraries. The following example shows one way to compile code for OpenCV:
g++ `pkg-config opencv --cflags` my_code.cpp -o my_code `pkg-config opencv --libs`
Step 3: The Game
The game is a basic 8 bit ping pong game. This is not the final version, there are still some bugs in the design and it has to be merged with the finger detection application. The purple fields on the screen are reserved for fingeres. At this time, the game is played with the keyboard (left player - Q and A, right player - O and L), and you can change speed (S) manually.
The game and the finger detection applications are designed separately and the idea is that the finger detection app detect the finger on the purple line, calculates the y coordinate and then sends that coordinate to game app witch then updates the position of players tile.
The game.zip file contains the source code that can be compiled with the folowing comand line:
g++ `pkg-config opencv --cflags` test2.cpp -o game `pkg-config opencv --libs`
and you can run the game with:
./game
Step 4: Finger Detection
With the web camera we are tracking multiple objects and calculating their coordinates on the screen. This part is not finished yet. What we done at this point is tracking of this simple blue objects. We can also detect and follow fingers, but you need to change bool variable calibrationMode on true (line 136). Then, when the program is built and executed, the slider window will appear, and first you need to change minimum and maximum HUE values to select the colors you want to detect. The detected object appears white on the screen. After that, with the you can reduce the noise with minimum and maximum values of SATURATION and VALUE (Picture 1).
If the calibrationMode is set to false, it will detect blue objects (Picture 2).
The finger.zip file contains the source code.
This is still work in progress so there are still bugs!
Project video: