How You Can Make a Fully Autonomous Self Driving Vehicle With Ai, Python and a Camera on a Budget

Step 1: Wait... What? How Can I Make One of These.

We have all seen cool videos of expensive cars like those from tesla and google driving by itself over public roads. While they are very impressive they are out of reach for most makers and DIY'ers. That is why I decided to make my own fully autonomous vehicle which uses some of the same advanced Ai and computer vision technologies that tesla, google and other companies use for their autonomous driving but for a fraction of the price!

In this project I am going to show you how you can turn any electric vehicle into a fully autonomous vehicle capable of driving by itself over real public roads! This works by using some advanced Ai detection and Ai learning technologies to make an Ai model predict steering and throttle responses according to the image coming from a camera. To see it work for real and a short introduction on the system, watch the video above.

Supplies needed:

1x vehicle with electric drive train. I use an electric kids ATV.

1x powerful desktop pc/laptop with Nvidia graphics. I use a GTX1080 with an Intel i7 Cpu.

1x mains power (gas) generator if you are using a pc. A laptop can run from its internal battery.

1x Xbox Kinect

1x wifi router for telemetry

1x Arduino Nano with USB cable

1x H bridge driver BTS7960

1x windshield wiper motor from a car

1x large sprocket (around 40 teeth)

1x small sprocket (around 13 teeth)

1x chain

2x safety buttons

1x large 10k potentiometer with metal shaft

Steel tubes to build a frame

A couple meters of wires.

Old ethernet cables work nicely

Some wood

Welding machine and angle grinder

Basic tools

Step 2: Let's Get Started With a Vehicle

First I would like to comment that this project is for experienced users who already have some knowledge of python, Arduino and electronics. This is needed because not everything I explain in this instructable will work right away. During this project there will be problems that you have to solve!

With that out of the way the first thing we need is a vehicle which is going to be the base for this project. To decide if your vehicle can be used for this project, you can use the list with requirements below:

• The vehicle must have an electric powertrain where the speed can be controlled using a pwm signal. Ideally the powertrain has a speed controller which accepts pwm signals with a range from 0V to 5V.

• The vehicle must be able to drive with a person on it.

• The vehicle must have a traditional (Ackermann) steering geometry.

• The vehicle should be robust. Something with a metal frame is the best. Also linkages like those for the steering system should not have any play.

• The vehicle should have some suspension to smooth out the ride for the camera.

• The vehicle should be large enough to fit large components like a pc and generator.

After having established the requirements I went looking for a vehicle. After looking on second hand sites I found this cheap electric chinees kid's ATV for only 50 euro's. This ATV is perfect since it has the perfect powertrain and steering geometry. It also has a perfect size for experimenting. If it starts driving by its own it wont break anything big or even worse kill someone.

With some research on second hand websites you will find the exact same ATV I am using since these things get made in the thousands in china. Or if you have a lot of money to spend you can buy a new one on amazon.

https://www.amazon.de/-/en/Cobra-Miniquad-Electric...

Above there are also some images of possible other vehicles to give you some inspiration. You can also opt to use a gas powered vehicle. This will however take some skills to make it work with my code.

Step 3: Make Some Hardware

Now that we have our vehicle it is time to get started with making the hardware for the autonomous system and adding those to the vehicle. To get an idea of what I did watch the video above. In this video I show what steps I took and how the hardware works.

The process for creating the hardware will depend on what vehicle you have chosen. But all vehicle types will need roughly the same steps listed below.

1. First start with making a metal frame on which the computer and gas generator will sit on. Use some wood to make the pc and generator fit on the frame.
2. Create a metal frame to mount the DC windshield wiper motor to the vehicle. Also add a sprocket to both the steering shaft and DC motor shaft. Make sure that these sprockets are mounted inline and in the center of the shafts. Finally add the chain to the sprockets. Ideally the chain does not have any slack to improve steering accuracy. I however was not able to achieve this on my vehicle.
3. Weld the potentiometer (which will be used to get the steering position) shaft to the steering system shaft of the vehicle. Make sure both shafts are inline with each other.
4. Add all the necessary electronics according to the component diagram above. This includes the wifi router, Xbox kinect (to decrease vibrations you could mount it on to of something soft like a sponge), pc and power generator. Also plug in all the necessary power, USB and ethernet cables.
5. Now wire all components together with electrical wires according to the schematic above. Below are some comments on the wiring diagram.
   1. The connections on the schematic are not clearly visible and not all connections are drawn. That is why I have also listed all connections in a separate table shown above. Use some common electrical knowledge to connect for example 5V to 5V off the Arduino and the GND of the Arduino, sensors and the motor controller together so that there is no floating ground.
   2. Make sure to add the selection switch as mentioned in the schematic. This switch will enable us to either read the signal coming from the throttle sensor which goes to the motor controller for learning the Ai or write to the motor controller for driving.
   3. Connect the output of the BTS7960 module to the windshield wiper motor. If during driving the vehicle turns the wrong way then switch the two wires from the windshield wiper motor.
   4. It is advised to add stop buttons to every actuator and power source. So a safety switch for the main battery, a switch between the main motor and speed controller and a switch between the BTS7960 and the windshield wiper motor.

After having completed the steps above you should have a fully assembled autonomous vehicle! If you want to see how I made it then watch the video above. Now let's get started with the Software!

Step 4: Embedded Software

The first thing we have to do with the software is to program the embedded computer (Arduino Nano). This device takes care of translating the digital signals coming over the USB connection from the Desktop pc to electrical signals for the vehicle actuators.

First we can check if the sensors and actuators from step 3 are correctly connected. To do this download the entire repo from Github found here:

https://github.com/sieuwe1/Advanced-Real-Time-Self...

Then navigate to this file and upload it to the Arduino with the Arduino software.

https://github.com/sieuwe1/Advanced-Real-Time-Self...

!!!please be caution when turning on the power on the vehicle!!!

This test script does not take into account the limits of your steering system. Start with a small delay like 1000 instead of the 5000. Then if the steering system does not actuate to its limits increase this delay. Also make sure that all the wheels of the vehicle are off the ground. Otherwise it could drive away without you :)

If during the test all actuators moved correctly, we can continue with uploading the final code to the Arduino. We need to upload a script which is already available from the Arduino software called "StandardFirmata". To upload this follow the steps described here:

https://www.instructables.com/Arduino-Installing-S...

After standard firmata is installed we can connect the Arduino to the Desktop pc using a USB cable. After that we can continue to step 5. Here we are going to do some Ai stuff!!

Step 5: Autonomous Software Installation

Now it is finally time for some software! If you want to know more about how the software works and how I created it then watch the video above! This will also give you some useful insights for the next steps.

First you want to make sure your environment is correct.

We want a Desktop pc with:

A decent Cuda compatible graphics card (I use a GTX1080)

At least 8GB of ram

A fairly recent i5 or i7 cpu

A SSD (we don't live in 2010)

And a clean ubuntu 18.4 LTS install! The software probably wont work on windows because windows is always broken!

If this is all checked then install all necessary libraries. The list of libraries and their correct version can be found here in the readme:

https://github.com/sieuwe1/Advanced-Real-Time-Self...

You can use the cameraTestRGB.py and cameraTestDepth.py scripts to check if the xbox Kinect is working correctly.

Tip: Use Anaconda environments to better organize your different python projects with different versions of libraries.

After all the modules are correctly installed it is time for some important calibrations!

Step 6: Vehicle Calibration

It is incredibly important that before we do any training we calibrate the vehicle. This is because we cant make any changes to this our vehicle after we have driven some training data. If we do make changes the data would be different which would make the Ai make false predictions.

We need to calibrate the following.

• The center position of the steering system sensor and the zero position of the throttle sensor

First we need to calibrate our sensors. This can be done by using the FindActuatorLimits.py script.

1. Before running the script however we have to make a small change to vehicleSerial.py. In this file we will change at line 5 "mapValues = True" to "mapValues = False".
2. After this start the FindActuatorLimits.py script.
3. While running the script rotate the steering system manually to its limits. So full left and full right.
4. Also manually give full throttle and zero throttle.
5. Do this a couple times to make sure all limits are reached.
6. After this stop the calibration program and write down the values: "minSteerSensorValue", "maxSteerSensorValue", "minThrottleSensorValue" and "maxThrottleSensorValue"
7. Now go back to vehicleSerial.py.
8. In this file enter the values you just wrote down.
9. As last make sure to change line 5 to True again like this "mapValues = True".

• The feedback controller for the Steering system and the feedback controller for the Throttle system

To perfectly tune the feedback controller we would need to dive deep into control systems which is an entire science of its own. We don't have time for that and because it is a simple p-controller we can just guess some values to get a good enough response. To do this we will use the calibrateControl.py script.

!!!Before you start lift the vehicle from the ground! During control calibration the motors will turn!!!

1. When the script starts it will keep turning the steering system and drivetrain to its limit and back again. While this movement is happening inspect how the movement is performed. It can either be performed too slow or too quick. The ideal speed is the speed that you would turn the steering system when you manually drive the vehicle.
2. After you have decided if it goes too fast or too slow, close the calibrateControl.py script and go to the control.py file. In this file change the "steeringAgressivity" and "throttleAgressivity" value. How higher this value is the faster the system will turn/give throttle. How lower it is how slower it will turn/give throttle.
3. Now run the calibrateControl.py script again and see how it performs. Keep playing with these numbers until the steering system and throttle system turn as fast as you would when manually driving the vehicle.
4. You could also try the "maxthrottleAgressivity" and "maxSteeringAgressivity" to get better results.

Don't get confused with the pid values that are on top of the control.py file. These are currently not used since I am still experimenting with them.

!!!Sometimes the Arduino does not get disconnected correctly resulting in the pins on the Arduino staying in the current state. This can result in the steering system or drivetrain to keep moving even after the script is closed. Always when closing a script that has movement also manually reset the Arduino using the reset button on the Arduino!!!

Step 7: Ai Training

Now that we have the entire vehicle sorted, it is time for some Ai training!

To train the Ai we are going to show the Ai how we drive the vehicle so that the Ai can later try to copy our behavior and drive just like us. This process is called behavioral cloning and is one very interesting method of making vehicles drive autonomously. This is because this method does not require any hard programmed rules and can achieve a human like driving level.

To start training we are going to use the driveTrainData.py script.

1. To run this script type "CUDA_VISIBLE_DEVICES=0 python3 driveTrainData.py".
2. Now type a name for the data folder. For example "run1".
3. Before you press enter, go and sit on your vehicle.
4. Now set the physical mode button on your vehicle (this button is explained in step 3) to learn mode. In this mode the signal from the throttle sensor is going to pin A1 on the Arduino.
5. Now press enter and start driving! It is very important that the driveTrainData.py script is only running when the vehicle is moving. When you want to stop driving, first close the driveTrainData.py script. After it has closed you can stop the vehicle.

It is important that you not only drive how you normally would but also drive to the edges off the road. This makes the Ai learn where the edge of the road is. For more tips on creating the best data, watch this useful video from the Donkeycar community.

The data folders you created are stored in the root of the Github repo. Navigate to this location and copy all your data folders to "Advanced-Real-Time-Self-Driving/tree/master/Ai_based/donkeycar/data" which is inside the Github repo. Make sure to copy all folders. If the folder structure gets changed the training script will give an error.

Now that the data folders are stored in the correct place it is time to start training. To do this simply run the following command.

"python3 Train.py --model=trained --type=linear"

After the training process is complete you will have a nice "trained.h5" file which will contain your valuable Ai network! You can test to make sure the network works correctly before trying it on your vehicle. This can save you from crashing into something :)

To do this change in SegmentationOnly.py, line 101 to the name of your Ai model. For example "driveModel = loadModel('trained.h5')". Also change in SegmentationOnly.py, line 167 to the path of a video file you want to test the model on.

Now run "CUDA_VISIBLE_DEVICES=0 python3 SegmentationOnly.py". You will see a window with a segmented area and a red line. This red line represent the predictions from the Ai drive model. The angle is of the red line is what direction it wants to go and the length of the red line is the speed it wants to go. Watch the red line and see if it predicts correct responses. If the Ai looks like it is doing some random stuff you probably have not trained it correctly.

To solve this you can try:

1. Drive more to gather more training data
2. Look at the youtube video above.
3. Recalibrate your sensors and drive new data. (After calibration you cant use your old data!)
4. Try one of my models (found in Github repo)
5. Try one of my data sets (found in Github repo)

Now that we have the Ai model trained and working correctly it is finally time to do some autonomous driving!

Step 8: The First Autonomous Drive!

It has probably taken you some time to get here. But now finally after some setbacks you can finally see your hard work pay off by seeing it drive autonomous!

Use the following steps to get it started:

1. Use the mode selection button on your vehicle to set it to Autonomous mode. In this mode the throttle sensor and motor controller wire is connected to pin 10 of the Arduino.
2. Change line 132 in driveAutonomous.py to use your model. For example "driveModel = loadModel('trained.h5')"
3. Now run "CUDA_VISIBLE_DEVICES=0 python3 driveAutonomous.py"
4. Now sit ready and press enter when the script says "Press enter to start vehicle"
5. After pressing enter the vehicle will start immediately with driving!
6. To stop the vehicle close the driveAutonomous.py script. Make sure to immediately reset the Arduino manually using the reset button on the Arduino to make sure the movement stops.

If it does not work immediately, don't worry! It also did not work the first time for me. The driveAutonomous.py script actually has a really useful debug option that writes everything the vehicle is doing into a debug folder. Use the --debug flag to turn this feature on. And if you cant figure it out you can always contact me or use the really helpful Donkeycar forum since a large part of the system is based on the work of Donkeycar.

I hope you liked this instructable! If you did please check out my other instructables and also check out my youtube channel for more projects like this!