Zynq Image Enhancement System

As you could probably make out from the title, the aim of this project is to make an Image Enhancement System using the ZYNQ ApSOC. More specifically, we wish to build a system that can clear out the fog from images or video. This system will take in poor conditions visual data as the input, process it using image enhancement techniques and then output the result.

The project was build and tested on the Digilent Zybo Board but other ZYNQ devices should work as well.

We will split this project in 3 parts :

1) INPUT = Input Image via Ethernet from Computer/Camera

2) PROCESS = Process the Image

3) OUTPUT = Output the Image via an HDMI interface

In a very counterintuitive fashion we will begin with the output part of the project (this will give us better debugging possibilities along the way) continue with the input and finish off with the processing part.

Step 1: Materials

To complete this project you will need :

HARDWARE

- any ZYNQ Board with HDMI and Ethernet should work / I'm using the Digilent Zybo

- USB A to micro B USB cable

- HDMI cable

- Ethernet cable

- Display with HDMI input

SOFTWARE

- Xilinx Vivado

- Xilinx SDK

Step 2: OUTPUT - VGA Controller Part 1

We will output our visual data using the HDMI port present on the board. The HDMI port is connected to the PL (Programmable Logic = FPGA) side of the ZYNQ and we will need to design a controller in VHDL for it. If you have ever designed a VGA controller you will find this very similar. The timings for HDMI and VGA are actually the same, in fact you can build on an existing VGA controller to obtain a HDMI controller.

For a better understanding of what is actually happening we will first design a VGA controller

We want to display at a resolution of 1920x1080.

The VGA controller is responsible for transmitting the pixel data (in RGB format) sequentially, pixel by pixel to the display. Outside of the actual display area area of 1920x1080 there are also some "border" areas, namely : front porch, back porch and retrace. The size in pixels of these areas are standard and specific to every resolution. These areas do NOT actually appear on the screen but they are mandatory and the color of the pixels in this area must be black. A valid question would be why are these extra areas needed. This question defies the purpose of this instructable but if you are curious i would encourage you to do further research online.

This is a good video explaining the VGA interface https://goo.gl/vfSw6o

In our case we want do display at a resolution of 1920*1080, and these are the timings :

Horizontal Display Area = 1920 pixels

Horizontal Fron Porch = 88 pixels

Horizontal Back Porch = 148 pixels

Horizontal Retrace =44 pixels

Vertical Display Area = 1080 pixels

Vertical Front Porch = 4 pixels

Vertical Back Porch = 36 pixels

Vertical Retrace = 5 pixels

( Here you can find timings for other resolutions http://goo.gl/hFNRVb )

So our actual resolution will be 2200 x 1125. We want 60 fps (frames per second) so our pixel clock will be 60*2200*1125 = 148.5 MHz. On the Zybo Board a 125 Mhz clock is provided. We will use an MMCM IP to generate the 148.5 MHz Pixel Clock we need.

Step 3: OUTPUT - VGA Controller Part 2

With the theoretical background from the previous step you should be able to design you own VGA controller. I will provide you with a Vivado project that does that but i advise you to at least try to make it on your own first.

Most VGA ports don't give you 8 bits per color channel per pixel (see image above) so you will need to adapt the design to the number of pins per color the Board provides (this is not a problem for the HDMI though).

The Design will paint the entire screen blue, except for the top left pixel which will be red. It should be noted that this project uses the constraints for the ZYBO Board. So if you want to run this project on another Board you should update the constraints file and adapt the number of pins per color.

Take a look at figure nr. 2. Remeber that while our VGA Controller outputs 5/6 bits per color, those bits get converted into one analogue signal for each color channel (Red, Green and Blue) before going through the cable.

Step 4: OUTPUT - HDMI Controller Part 1

Now that we know how the VGA controller works and we have a working design we can continue with the HDMI controller. The HDMI controller actually will use all the code we developed in the VGA controller. The HDMI and VGA use the same timings and the same signals. The difference appears on the ouput pins.

While VGA uses one wire for each color and transmits an analogue signal accross it , HDMI transmits the data digitally 1 bit at a time for each color and uses differential signaling. Differential signaling means that for each bit the HDMI has 2 pins with one the opposite of the other. So if we would want to transmit a signal '1' we would transmit '1' on a wire and '1' negated on the other wire. This ensures signal integrity and you can read more about it here https://goo.gl/6CPCzB . We have one of these channels for each color, RED, GREEN and BLUE and one for the clock. Because of the specifics of differential signaling the signals we are sending via hdmi must be DC balanced that means that the number of 1's and 0's must pe roughly equal in a certain window of time. In order to accomplish this we will use 8b/10b encoding. You can learn a lot about how differential signaling and 8b/10b encoding works from the DVI specification here http://goo.gl/hhh8Ge ( DVI and HDMI use the same video signals).

Step 5: OUTPUT - HDMI Controller Part 2

Enough theory, lets get to our project. While in the VGA Controller we got away with a 148.5 MHz clock, here we will have to provide 10 times that frequency because we want to transmit 8 bits for each color and using the 8b/10b encoding that translates to 10 bits per pixel and 10*148.5MHz = 1485MHz. That is a huge frequency that cannot be obtained on the Zybo Board. Fortunately we got a few tricks up our sleeve. We can manage 5*148.5MHz = 742.5MHz and we will use an OSERDES (serializer) IP to transmit data both on the rising and falling edge of the 742.5Mhz clock, so we will actually get data transmitted at 1485MHz. Vivado will give us some timing warnings and you could always go for a lower resolution with a smaller clock, but since it works, we don't really mind it for now(the warnings are related to the fact that the clock buffers do not officially support frequencies higher than 464MHz).

So what we need to do is to encode the data from our VGA Controller output in 8b/10b format and then serialize it as mentioned above. We will also need to add another MMCM to the project to generate the 742.5MHz clock for the serialization.

I attached bellow the vhdl files for the encoder and serializer. You must first encode the RGB channels and then serialise them.

Example for the red channel :

TMDS_encoder_RED : TMDS_encoder

port map(clk148, red_channel_8bits, c_red, video_on, encoded_red_10bits);

Serialiser_RED : Serialiser10_1

port map(clk148, clk742, encoded_red_10bits, reset, red_serial_1bit);

The "c" input to the TMDS_encoder is "00" for red and green and "vsync & hsync" for blue (this is part of the DVI specification http://goo.gl/hhh8Ge ).

Step 6: Displaying Images From RAM

The purpose of the the HDMI controller is to display the processed images. Now, with the controller implemented and ready to go we should think about feeding this controller with data. Given that a lot of the image enhancement process will take place in the PS (Processing System = ARM Processor) and the images resulted will reside in the DDR RAM. So we need a way to get the data from the RAM to the HDMI controller.

In order to accomplish this you will need 3 IPs :

1) VDMA (Video Direct Memory Access)

2) VTC (Video timing Controller)

3) Stream to Video Out (we will call it S2VO from now on)

S2VO will actually provide an RGB 24BIT signal to the output and the needed HSYNC and VSYNC signals. So we can leave that part of the HDMI controller out.

You should add these IPs to your design, configure them and make the proper connections.

Finally you should get something resembling the schematic above.

Step 7: OUTPUT - SDK END

With all the hardware set up and ready to go we must now build the software in the PS. We will export the hardware and the bitstream and launch the SDK.

1) File -> Export -> Export Hardware -> Check Include Bitstream and Press OK

2) File -> Launch SDK

In the SDK create a new application project.

3) File -> New -> Application Project

4) Choose a name for your project and press Next

5) Select the "Hello World" template and press Finish

The application in the SDK will need to program the VDMA . There are some standard functions used in order to accomplish this (i will go into details when i have time).

In order to test our design we will use the SDK Restore (Xilinx Tools -> Dump/Restore) feature to put an image into the DDR RAM memory and display it using our HDMI Controller. You can load the image anywhere you want (except some small restricted areas in the beginning of the memory). For our example we chose address 16777216 and the file size 8294400 = 1920*1080*4 (4 channels = RGB + alpha).

It Works !

To be continued