Introduction: Thermal Imaging Camera

This instructable is going to show you the step by step process to replicate the Thermal Imaging Camera project.When everything is finished you will be able to see an infrared image with a slight contour from the visible spectrum camera on the Monitor. Basicaly we are trying to replicate the functionality and versatility of a commercial thermal imaging camera which costs roughly 1000 euros or more.

Below we will present you the steps to replicate our project and also a few Important tips.

For this instructable you will need :

  • ZedBoard Zynq-7000 ARM/FPGA SoC Development Board with a successfully installed distribution of Linux (Arch, Linaro,Xillinux)/Zybo Zynq7000
  • Flir Lepton Development kit (Breackout board + Lepton Longwave Infrared (LWIR) imager )
  • USB camera or any other camera.
  • One Raspberry Pi 3 b with Raspian installed and OpenCV
  • microSD card of 16 to 32GB(Linux compatible ) you should check on this link: http://elinux.org/RPi_SD_cards
  • USB OTG cable(supplied with the zedboard), Wall plug Powered USB Hub,Keyboard,Mouse,Monitor with VGA input

Step 1: ZedBoard /Zybo Setup

First of all you will have to install the Xillinux distribution for Zedboard/Zybo . A fully documented tutorial is available on Xillybus website http://xillybus.com/xillinux

Steps:

1.Prepare the SD card

You will have to download the Xillinux Image and the Boot partition kit provided on the website ( http://xillybus.com/xillinux )

Using Win32 Disk Imager or USB Image tool load the microSD card with the image.. The detailed process is explained in the Getting Started guide for Zynq (Zedboard) section 3.6 page19


Link to theGetting Started guide for Zynq (Zedboard)

http://xillybus.com/downloads/doc/xillybus_getting...

After the process has been finished you will have on the microSD card 2 partitions one which will contain the boot files of 15Mb and another one that is hidden in windows with the format ext4 ( linux specific) that contains the linux image.

For the momment on the 15MB partition you will find just a uimage file.

Step 2: Generating the Bitstream

For this step you can chose Vivado or ISE for generating the bitstream file.

Here you will have to use the previously downloaded zip file for zedboard xillinx-eval-zedboard-1.3c and

for zybo xillinx-eval-zybo-1.3c

For this part we have chosen Vivado.

-> Create a new folder and unzip xillinx-eval-zedboard-1.3c or xillinx-eval-zybo-1.3c

-> open Vivado-> Tools-> Run Tcl script

->go in the newly created folder-> verilog folder and select to open xilidemo-vivado.tcl (You can also chose the VHDL file)

After the project initialization click on the Generate Bitstream (on the left in the Flow Navigator->Program and Debug )and wait for the process to finish.

When finished the generated bitstream will be found in the folder impl_1 on the path Your_NewFolder\xillinux-eval-zedboard-1.3c\verilog\vivado\xillydemo.runs\impl_1

Step 3: Copy All the Boot Files on the 15MB Partition

In this step you will have to copy all the boot files to the 15Mb partition on the microSD card.

The uImage file is already present on the target partition we will add the bitstream file (xillydemo.bit) from the previous step and also the two files contained in the bootfiles directory (\xillinux-eval-zedboard-1.3c\bootfiles) the boot.bin and the devicetree.dtb.


If you ever get stuck we recommend you follow the above mentioned guide http://xillybus.com/downloads/doc/xillybus_getting...

Step 4: Set the Jumpers on the Board

Section 4.1 of the Getting Started guide for Zynq (Zedboard) .http://xillybus.com/downloads/doc/xillybus_getting... deals with the prooper set up of the jumpers.

Step 5: Resize the MicroSD Partition Size

After the first boot it is recommended to extend you working partition.

Because we have written the image on the sd card the partition downsized to 1.6GB and there is no reason to not use the full space.

The commands used are :" df -h" to show the available partitions ,"fdisk /dev/mmcblk0" after executing the fdisk command press 'd' for deleting the partition then '2' for selecting the partition to be deleted .After that press 'n' to create a new partition and you can use the default settings pressing 4 times the enter key. After this press 'w' to save the changes then use the command "shutdown -r now" to rebot .

After rebooting type in the command "resize2fdisk /dev/mmcblk0p2" and there you go your partition size is at full lenght.

More details can be found in the Getting Started guide for Zynq (Zedboard) section 4.4 .http://xillybus.com/downloads/doc/xillybus_getting...

Step 6: Update and Upgrade the Distribution

Warning there is limited to no support for the XIllinux distribution. Only recently we have succeeded in upgrading to a newer version.

Open the Terminal window: type the commands "sudo apt-get update" ,"sudo apt-get upgrade".

After the upgrade follow the steps:

1. During start-up you will press "Ctrl+alt+f1" to be able to use the root user ,create a new one and change the root password

2.Add a user using the command "adduser"

3. Change the root password using the command "sudo passwd root".

Step 7: Installing OpenCV on the ZedBoard or Zybo

Use the documentation provided on the OpenCV website , in the Linux section. http://docs.opencv.org/2.4/doc/tutorials/introduct...

We recommend installing an older version ex: 2.4.12.

Step 8: Setup the Raspberry Pi

1.Install Noobs on the Raspberry Pi-> https://www.raspberrypi.org/help/noobs-setup/

2.Enter in the Raspberrypi 3 configuration menu (follow the tutorial https://www.raspberrypi.org/documentation/configur... )

3. Activate the SPI and I2C

->When you are in the configuration menu go to "Advanced Options"

->Select SPI and I2C and then reboot

4.Pure Engineering provides us with the Driver for the FLIR sensor.

-> Open a terminal window

->We will install the following dependency with the command "sudo apt-get install qt4-dev-tools"

-> Create a new folder and "cd" your way in, from the Terminal.

->Downoad from the github the modules for the FLIR using the command "git clone https://github.com/groupgets/LeptonModule" (#FLIRMODULE)

5.Connect to the Raspberry PI 's GPIO pins the FLIR. The exact pinout for I2C and SPI is presented in the images section. Do not forget the Power and GND pins.

6."cd" your way into the directory in which were downloaded the FLIR modules from the git hub

7."cd" your way in the folder "LeptonModule-master" -> then the folder "LeptonSDKEmb32PUB" and run the command "make" [this will make the Lepton SDK ]

8. "cd" your way back in the “raspberrypi_video” directory and run the command "qmake && make" [this step compiles the example code] [#SAME_TERMINAL_WINDOW]

9. Check the connections to the FLIR sensor <->Raspberry PI and make sure they are snug.

10. Input the command in the terminal window "sudo ./raspberrypi_video" [#SAME_TERMINAL_WINDOW]

11. You should have a window showing a raw image from the FLIR sensor.

Step 9: Install OpenCV on the Raspberry PI

Install OpenCV 2.4.12 on the Raspberry Pi

1.Install the libraries and dependencies for OpenCV. Always keep the Linux updated.

sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade && sudo apt-get autoremove
sudo apt-get install build-essential cmake
sudo apt-get install qt5-default libvtk6-dev
sudo apt-get install zlib1g-dev libjpeg-dev libwebp-dev libpng-dev libtiff5-dev libjasper-dev libopenexr-dev libgdal-dev
sudo apt-get install libdc1394-22-dev libavcodec-dev libavformat-dev libswscale-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev yasm libopencore-amrnb-dev libopencore-amrwb-dev libv4l-dev libxine2-dev
sudo apt-get install libtbb-dev libeigen3-dev

2. Download and unzip the archive somewhere on the disk: https://github.com/Itseez/opencv/archive/3.1.0.zip

3. Create a directory inside the folder where you extracted the OpenCV, call it build.

4. Open the Terminal and write 

cd Path/opencv/build
cmake -DWITH_QT=ON -DWITH_OPENGL=ON -DFORCE_VTK=ON -DWITH_TBB=ON -DWITH_GDAL=ON -DWITH_XINE=ON -DBUILD_EXAMPLES=ON ..
make -j4
sudo make install
sudo ldconfig

Now the OpenCV is installed.

Step 10: Camera Setup 1 of 2

1.The frame

Decide from what material are you going to build the camera frame and place the FLIR Lepton sensor beneath the visible camera on the frame. We have chosen the support from the visible camera which is adequate for sustaining the FLIR Lepton sensor and has a flush vertical position.

2.The wiring

Solder some female connectors to an UTP cable and on the other side some male connectors.Drill a hole on the back of the case and run the two cables (USB from the visible camera and the custom made cable for the FLIR sensor)

3.The case

For the case we have chosen a plastic material.Place the frame with the two cameras inside your designed case and glue it in place.

Step 11: Camera Setup 2 of 2 and Raspberry PI Connection

For the moment we have the software working just in Raspberry Pi 3. The software can be easily adapted for any Linux platforms just if the SPI Module is working properly. When we will done the streaming software (TCP transmission) we will move everything on Zybo. We couldn't find any sources about how to use the SPI Module from Zybo on Linux so the other solution was to use the Raspberry Pi as "adapter" from SPI to LAN. The software developed on Raspberry Pi will be adapted to get sensor data from LAN instead of using SPI Module from Zybo.

Here you can find the wiring diagram, software diagram and the working software.

Attachments

Step 12: Streaming the Image to the ZedBoard/Zybo

Dear all,

Here is the project in its final state , we have worked on it until before the presentation at the Digilent design contest 2016 edition . Attached you will find the complete project together with documentation and the source files. I hope it helps you in your endeavours.

Good luck and take care !

Attachments

Step 13: