Introduction: FPGA Robotics Vision for Anyone

Picture of FPGA Robotics Vision for Anyone

Are you like me interested in technology and always wanted to use one of those heavy chips called FPGA's? Are you interested in vision and robotics?

My goal was to bring FPGA vision to the people, accessible to everyone. I decided to build my own basic vision system. Not based on some FPGA manufacturers expensive IP and hardware. No! Just by using a camera, LCD display that can be found anywhere on Ebay or chinese webshops.

I used an affordable FPGA from Xilinx on a VERY affordable $31 development board...the only thing required was lots of VHDL code!

Why did I use an FPGA to do this? Most people use a Microcontroller right? And there are options out there that give your robot out-of-the-box basic vision for a fair price like the Pixy CMUCAM5. I own one, it is based on 2 ARM microcontrollers. I used an FPGA because of its tremendous power to do all kinds of tasks in parallel making it very useful for a task like Vision. And because I think there are more peope like me who want to use an FPGA but did not know where to start.

The result is a real-time 30fps imaging system with basic color detection. This is called "tresholding". What it does is compare the QVGA images coming from the camera at 30fps with a pre-defined "mask" for the R, G and B components. The system "spots" the pre-defined color and shows this on the 320x240 (QVGA) LCD. Pixels complying with the "mask" condition are shown white and those who not comply stay black.

Right now (in the VHDL code) this "mask" is a bright almost white color. This means the R,G and B components are almost at their maximum.

I also implemented "template matching". This means you can compare a pre-stored camera image to an external FIFO memory (design of the board is included) with the real-time images coming from the camera and see the differences real-time on the LCD! I will bring this later to Instructables.com.

Feel free to use my system to experiment and maybe implement your own algorithms.

Step 1: Shopping for the Parts and Stuff

Picture of Shopping for the Parts and Stuff

The Bill of Materials is not so large. I used an FPGA board with a Xilinx FPGA and it was a Spartan-6 (those are affordable).

I have tried other OV7670 modules with screw-on lenses (still without FIFO) but have seen issues with those: they gave me green colored images.

That is why I used the version with the miniature camera (no focus lense) and ZIF socket. It costs slightly more but gave me no problems.

Step 2: Setting Up the Xilinx ISE Environment

The installation of Xilinx ISE is straight forward. Switch off your antivirus, it speeds up the installation.

The setup of the project should not be required if you just open the project from the ZIP file I added to Step 1.

However might there be any issue please refer to the User Manual of the Ebay Xilinx board I recommended in Step 1. I attached it to this step.

It describes in great detail how to install ISE and also how to setup your projects tailored to this FPGA board.

Step 3: Wiring the Camera and TFT to the FPGA Board

I presume in this step that you use the same Spartan 6 board from Ebay which I mentioned earlier in Step 1. Connections present on the Spartan board are not shown (CLK in, leds, SW1, SW2 etc).

Connections mentioned here you need to make between FPGA board, TFT and Camera using the Dupont wires.

In the Xilinx project folder the .UCF file can be found which contains the complete FPGA board connections , use this if you have another Spartan6 board or you want to make your own board.

See the labels with pin nrs on the Spartan board for pin numbers (e.g: P95)

Camera databus

OV7670_DATABUS[7] TO SPARTAN6 XC6SLX9 FPGA PIN P95

OV7670_DATABUS[5] TO SPARTAN6 XC6SLX9 FPGA PIN P98

OV7670_DATABUS[6] TO SPARTAN6 XC6SLX9 FPGA PIN P94

OV7670_DATABUS[3] TO SPARTAN6 XC6SLX9 FPGA PIN P100

OV7670_DATABUS[4] TO SPARTAN6 XC6SLX9 FPGA PIN P97

OV7670_DATABUS[2] TO SPARTAN6 XC6SLX9 FPGA PIN P99

OV7670_DATABUS[1] TO SPARTAN6 XC6SLX9 FPGA PIN P102

OV7670_DATABUS[0] TO SPARTAN6 XC6SLX9 FPGA PIN P101

Camera control signals

OV7670_HREF TO SPARTAN6 XC6SLX9 FPGA PIN P87

OV7670_PIXELCLOCK TO SPARTAN6 XC6SLX9 FPGA PIN P93

OV7670_SCL TO SPARTAN6 XC6SLX9 FPGA PIN P85

OV7670_SDA TO SPARTAN6 XC6SLX9 FPGA PIN P84

OV7670_VSYNC TO SPARTAN6 XC6SLX9 FPGA PIN P88

OV7670_XCLK TO SPARTAN6 XC6SLX9 FPGA PIN P92

TFT screen databus

TFT_DATA[15] TO SPARTAN6 XC6SLX9 FPGA PIN P27

TFT_DATA[14] TO SPARTAN6 XC6SLX9 FPGA PIN P29

TFT_DATA[13] TO SPARTAN6 XC6SLX9 FPGA PIN P24

TFT_DATA[12] TO SPARTAN6 XC6SLX9 FPGA PIN P26

TFT_DATA[11] TO SPARTAN6 XC6SLX9 FPGA PIN P22

TFT_DATA[10] TO SPARTAN6 XC6SLX9 FPGA PIN P23

TFT_DATA[9] TO SPARTAN6 XC6SLX9 FPGA PIN P17

TFT_DATA[8] TO SPARTAN6 XC6SLX9 FPGA PIN P21

TFT_DATA[7] TO SPARTAN6 XC6SLX9 FPGA PIN P15

TFT_DATA[6] TO SPARTAN6 XC6SLX9 FPGA PIN P16

TFT_DATA[5] TO SPARTAN6 XC6SLX9 FPGA PIN P12

TFT_DATA[4] TO SPARTAN6 XC6SLX9 FPGA PIN P14

TFT_DATA[3] TO SPARTAN6 XC6SLX9 FPGA PIN P10

TFT_DATA[2] TO SPARTAN6 XC6SLX9 FPGA PIN P11

TFT_DATA[1] TO SPARTAN6 XC6SLX9 FPGA PIN P8

TFT_DATA[0] TO SPARTAN6 XC6SLX9 FPGA PIN P9

TFT screen control signals

CS_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P32

EN_BACKLIGHT TO SPARTAN6 XC6SLX9 FPGA PIN P35

ILI9325_READY TO SPARTAN6 XC6SLX9 FPGA PIN P41

RD_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P33

RESET_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P40

RS_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P30

WR_ILI9325 TO SPARTAN6 XC6SLX9 FPGA PIN P34

Step 4: Explenation of the VHDL Code

Picture of Explenation of the VHDL Code

VHDL code is not like C code. Actually VHDL is a description language, with this language you can "describe" how the FPGA chip should behave. It's basically like drawing a schematic but now you do it in words.

Toplevel

Have a look at the picture attached to this step, this is called Toplevel. On the toplevel all individual code blocks can be seen and I will describe shortly what their functions are.

POWER_ON_RESET_CONTROLLER

This block takes care that all the others are properly reset when the power is applied to the system. It does so by waiting an amount of time using a state-machine and while waiting it makes it's output NOT_POR_RESET low. This signal is connected to all other blocks and resets them / prevents them from starting until this reset pulse is over (going to HIGH level).

Blinkled

Just my way of hello-world, I use this to let my board indicate its alive (by blinking leds)

clock_divider

Divides the clock by 2 and feeds this to the OV7670 because the OV7670 cannot use the 50MHz from the fpga board (too high).

datasplitter

Acts as a bidirectional databus buffer with 3state option. Not really used yet, the TFT is only written to but it is possible to also read from the TFT registers.

ili9325_16bitcameradata

This block first configures the TFT it is connected to (right after power on reset is finished). For instance it sets the TFT into RGB565 mode. After that it goes into write mode and all it does is constantly update the TFT view (showing either camera images or tresholded images. When this block is done configuring it signals to the next block that it can start it's configuration:

OV7670_sccb_initialisation

This block took a lot of work. The Omnivision camera (OV7670) needs to be configured after power-up. For example we need to set it's output to RGB565 mode (meaning the R-factor of a pixel is 5 bits, G is 6 and B is 5 again). I chose this to make it match with my ILI9325 TFT which has the same feature. Sccb is very similar to I2C (I-square-C, protocol invented by Philips a long time ago). But it has its differences which took me many many hours to get right!

image_tresholding

Last but not least. This is where the actual magic happens. The block gets images from the OV7670 camera (driven by the camera's HREF and VSYNC lines which together indicate a start + end of frame (image)). Inside the block the comparison for each pixel is done (realtime!) with the characteristics of the pixel range we look for (in this case bright pixels). SW1 is there to choose what to see on the TFT: normal camera images or the tresholded images. The image_treshold block also sends the VSYNC and HREF lines of the camera to the ili9325 block which is required to drive and synchronise the TFT to show the images.

So this was the basic description of all the blocks, those who want more details I invite to have a look in the VHDL codefiles for all these blocks.

Step 5: We Have a Visual!

Picture of We Have a Visual!

It is now time to load the BIN file to your FPGA board (or use the .MCS file which is the FLASH memory config file, the difference is that when you use this then after power down and power up again the program is reloaded and when flashing the BIN file this is not the case). Loading the FPGA with its configuration file is done via iMpact from Xilinx. If this is not familiar to you then please have a look at the user manual of the FPGA board which I attached in Step 2.

I hope you enjoyed my Instructable and you also have a visual on your TFT screen like the example picture with this step shows! Here you see the result of very bright lights which are "detected" by the FPGA program: the treshold is set to very intense R,G and B levels in the pixels that the camera captures. The result is this black/white (tresholded) image.

I hope I will see lots of people intrigued by the power of an FPGA, picking up my FPGA vision and bring it to the next level!

Comments

BobA48 (author)2016-04-27

Hi thank you so much for sharing your work !

Can you explain me why did you choose a bare camera (with no fifo) pros and cons.

Dennisv15 (author)BobA482016-04-27

Hi!

You're welcome!

My reasons in the first place not to use the FIFO variant were that an FPGA is so fast that it doesnt need the FIFO. Second reason is that I had good working C-code examples (STM32 ARM processor) of the OV7670 with no FIFO

However, I am going to post a 2nd project soon. Here I do template matching. And here I do use the same FIFO. Only with 1 big difference: the camera's sold on Ebay with the FIFO included ALWAYS store their frames in the FIFO and the user has to load them from that FIFO. My implementation (with a custom PCB design for the FIFO) uses the FPGA to route the Camera databus either directly into the FPGA or through it to the FIFO. The reason is I use the FIFO to take a "snapshot". Then The FPGA gets the snapshot out of the FIFO and compares this with the actual (real-time) camera view and indicates all pixel differences on the TFT.

So its a long story.

You could however adapt the current tresholding example in such a way that the FIFO variant camera can be used?
I would not recommend it because you can buy this for about 5 dollar this FIFOless camera!

Hope this answers your question.

Cheers
Dennis

BobA48 (author)Dennisv152016-04-27

Thank you for your fast answer. Actually I have and hy28b (ili9325) and

OV7725 with FIFO bought on ebay. My goal was to do a similar project to yours with my FPGA DE0 nano board playing arround image processing. I am still working on how to init the ili9325, this driver drives me crazy ! Im trying to establish the right init sequence on my stm32 discovery board for the moment. Currently I can read the device part number at address 0x00, so my SPI seems well configured but i can't go further. I looked at your vhdl code to get the right init sequence and you set registers that are not specified in the datasheet for example 0xe5 and 0x00 wich is read only according to the datasheet. I am a little bit lost.

Dennisv15 (author)BobA482016-04-27

Hi! I can imagine you are a bit lost. I was too. What goes for the ILI9325 also goes for the OV7670: disatrous datasheets. For the ILI I looked at many examples mostly from china. Yes indeed there are registers written that are supposed to be readonly. I recommend you use exactly my sequence this does work.

If you want to try with STM32 I have a project for that.

Send me a personal message and I'll attach the C version.

About This Instructable

3,247views

38favorites

License:

More by Dennisv15:FPGA Robotics Vision for anyoneAcrylic sheet RGB glowing Weather station with indoor/outdoor/forecast using ESP8266
Add instructable to: