Introduction: FPGA Robotics Vision for Anyone
Are you like me interested in technology and always wanted to use one of those heavy chips called FPGA's? Are you interested in vision and robotics?
My goal was to bring FPGA vision to the people, accessible to everyone. I decided to build my own basic vision system. Not based on some FPGA manufacturers expensive IP and hardware. No! Just by using a camera, LCD display that can be found anywhere on Ebay or chinese webshops.
I used an affordable FPGA from Xilinx on a VERY affordable $31 development board...the only thing required was lots of VHDL code!
Why did I use an FPGA to do this? Most people use a Microcontroller right? And there are options out there that give your robot out-of-the-box basic vision for a fair price like the Pixy CMUCAM5. I own one, it is based on 2 ARM microcontrollers. I used an FPGA because of its tremendous power to do all kinds of tasks in parallel making it very useful for a task like Vision. And because I think there are more peope like me who want to use an FPGA but did not know where to start.
The result is a real-time 30fps imaging system with basic color detection. This is called "tresholding". What it does is compare the QVGA images coming from the camera at 30fps with a pre-defined "mask" for the R, G and B components. The system "spots" the pre-defined color and shows this on the 320x240 (QVGA) LCD. Pixels complying with the "mask" condition are shown white and those who not comply stay black.
Right now (in the VHDL code) this "mask" is a bright almost white color. This means the R,G and B components are almost at their maximum.
I also implemented "template matching". This means you can compare a pre-stored camera image to an external FIFO memory (design of the board is included) with the real-time images coming from the camera and see the differences real-time on the LCD! I will bring this later to Instructables.com.
Feel free to use my system to experiment and maybe implement your own algorithms.
Step 1: Shopping for the Parts and Stuff
The Bill of Materials is not so large. I used an FPGA board with a Xilinx FPGA and it was a Spartan-6 (those are affordable).
- You need to go to http://www.xilinx.com and register an account. After that download the Xilinx ISE webpack Version 14.7
- Download the ZIP file attached to this step, it contains the project code including all VHDL files
- a real nice guy from China I know called “Adam” sells a good and affordable FPGA board Xilinx-XC6SLX9-based board like the Xilinx-XC6SLX9-Spartan-6-FPGA-Core-Board-Nano-Size on ebay
- You will also need a Xilinx programmer, the original is very expensive but a clone can be obtained for $31,-
- An 2.8" TFT screen with ILI9325 chipset like this one (I used it in 16bit mode RGB565 so be sure that the one you buy supports this)
- An OV7670 camera board with NO fifo mounted (bare-camera board)
- A set of so-called Dupont cables like these to wire it all together
I have tried other OV7670 modules with screw-on lenses (still without FIFO) but have seen issues with those: they gave me green colored images.
That is why I used the version with the miniature camera (no focus lense) and ZIF socket. It costs slightly more but gave me no problems.
Step 2: Setting Up the Xilinx ISE Environment
The installation of Xilinx ISE is straight forward. Switch off your antivirus, it speeds up the installation.
The setup of the project should not be required if you just open the project from the ZIP file I added to Step 1.
However might there be any issue please refer to the User Manual of the Ebay Xilinx board I recommended in Step 1. I attached it to this step.
It describes in great detail how to install ISE and also how to setup your projects tailored to this FPGA board.
Step 3: Wiring the Camera and TFT to the FPGA Board
I presume in this step that you use the same Spartan 6 board from Ebay which I mentioned earlier in Step 1. Connections present on the Spartan board are not shown (CLK in, leds, SW1, SW2 etc).
Connections mentioned here you need to make between FPGA board, TFT and Camera using the Dupont wires.
In the Xilinx project folder the .UCF file can be found which contains the complete FPGA board connections , use this if you have another Spartan6 board or you want to make your own board.
See the labels with pin nrs on the Spartan board for pin numbers (e.g: P95)
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P95
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P98
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P94
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P100
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P97
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P99
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P102
OV7670_DATABUS TO SPARTAN6 XC6SLX9 FPGA PIN P101
Camera control signals
OV7670_HREF TO SPARTAN6 XC6SLX9 FPGA PIN P87
OV7670_PIXELCLOCK TO SPARTAN6 XC6SLX9 FPGA PIN P93
OV7670_SCL TO SPARTAN6 XC6SLX9 FPGA PIN P85
OV7670_SDA TO SPARTAN6 XC6SLX9 FPGA PIN P84
OV7670_VSYNC TO SPARTAN6 XC6SLX9 FPGA PIN P88
OV7670_XCLK TO SPARTAN6 XC6SLX9 FPGA PIN P92
TFT screen databus
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P27
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P29
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P24
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P26
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P22
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P23
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P17
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P21
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P15
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P16
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P12
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P14
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P10
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P11
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P8
TFT_DATA TO SPARTAN6 XC6SLX9 FPGA PIN P9
TFT screen control signals
CS_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P32
EN_BACKLIGHT TO SPARTAN6 XC6SLX9 FPGA PIN P35
ILI9325_READY TO SPARTAN6 XC6SLX9 FPGA PIN P41
RD_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P33
RESET_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P40
RS_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P30
WR_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P34
Step 4: Explenation of the VHDL Code
VHDL code is not like C code. Actually VHDL is a description language, with this language you can "describe" how the FPGA chip should behave. It's basically like drawing a schematic but now you do it in words.
Have a look at the picture attached to this step, this is called Toplevel. On the toplevel all individual code blocks can be seen and I will describe shortly what their functions are.
This block takes care that all the others are properly reset when the power is applied to the system. It does so by waiting an amount of time using a state-machine and while waiting it makes it's output NOT_POR_RESET low. This signal is connected to all other blocks and resets them / prevents them from starting until this reset pulse is over (going to HIGH level).
Just my way of hello-world, I use this to let my board indicate its alive (by blinking leds)
Divides the clock by 2 and feeds this to the OV7670 because the OV7670 cannot use the 50MHz from the fpga board (too high).
Acts as a bidirectional databus buffer with 3state option. Not really used yet, the TFT is only written to but it is possible to also read from the TFT registers.
This block first configures the TFT it is connected to (right after power on reset is finished). For instance it sets the TFT into RGB565 mode. After that it goes into write mode and all it does is constantly update the TFT view (showing either camera images or tresholded images. When this block is done configuring it signals to the next block that it can start it's configuration:
This block took a lot of work. The Omnivision camera (OV7670) needs to be configured after power-up. For example we need to set it's output to RGB565 mode (meaning the R-factor of a pixel is 5 bits, G is 6 and B is 5 again). I chose this to make it match with my ILI9325 TFT which has the same feature. Sccb is very similar to I2C (I-square-C, protocol invented by Philips a long time ago). But it has its differences which took me many many hours to get right!
Last but not least. This is where the actual magic happens. The block gets images from the OV7670 camera (driven by the camera's HREF and VSYNC lines which together indicate a start + end of frame (image)). Inside the block the comparison for each pixel is done (realtime!) with the characteristics of the pixel range we look for (in this case bright pixels). SW1 is there to choose what to see on the TFT: normal camera images or the tresholded images. The image_treshold block also sends the VSYNC and HREF lines of the camera to the ili9325 block which is required to drive and synchronise the TFT to show the images.
So this was the basic description of all the blocks, those who want more details I invite to have a look in the VHDL codefiles for all these blocks.
Step 5: We Have a Visual!
It is now time to load the BIN file to your FPGA board (or use the .MCS file which is the FLASH memory config file, the difference is that when you use this then after power down and power up again the program is reloaded and when flashing the BIN file this is not the case). Loading the FPGA with its configuration file is done via iMpact from Xilinx. If this is not familiar to you then please have a look at the user manual of the FPGA board which I attached in Step 2.
I hope you enjoyed my Instructable and you also have a visual on your TFT screen like the example picture with this step shows! Here you see the result of very bright lights which are "detected" by the FPGA program: the treshold is set to very intense R,G and B levels in the pixels that the camera captures. The result is this black/white (tresholded) image.
I hope I will see lots of people intrigued by the power of an FPGA, picking up my FPGA vision and bring it to the next level!
Participated in the
Robotics Contest 2016