Introduction: Artificial Intelligence to Localize Thanks to Echos As Bats

For the fun I wanted to design from scratch a home made robot that would be able to move autonomously inside my house.

Firstly I worked on mechanical, electronic and and software in order to be able to control the moves (for instance DC motor rotation or use of Inertial Measurement Unit). (Motion Control)

Even if this prototype need improvements, it is reliable enough to go from one determined place to a desired one.

But for the robot to run autonomously, the robot must be able to regularly answer the question : where am I ?

According to the amount of experiences we can find on internet,I should have tried with image recognition but I choose to try a other lead.

Echo sensors are frequently used for obstacle avoidance. I search over Internet about using those sensors for robot localization inside a space but I got nothing.

I wanted to explore the possibility to use echos to do so and this instructable is about work.

Step 1: Let's Start With an Overview on Autonomous Motion

The 3 main questions are:

  1. where am I ?
  2. where am I going ?
  3. how to get there ?

This instructable will ignore the second one and focus on the first one (red dots rectangle). It is important to understand that if the robot position is known precisely at the instant t0 each move, even with a high performance motion control, will add some imprecision in the futur location.

So the system to design must be enslaved to some feedbacks from real world in order to keep theoretical and real points of view as close as possible.

Step 2: Let's Zoom on the Perception System

The idea is quite simple

  1. set the robot in a predefined orientation using a compass

  2. use echo sensors to get a set of distances 360° around the robot

  3. build a 2D image with the set of echoes

  4. compare this image with a set of previously learned ones

  5. determine the current robot location

Step 3: Cartography Principles

Space is divided in 30 x 30 cm squares.

Each square is divided into 9 sub-squares of 10x10cm. For each sub-square an integer defines the characteristics (roughly occupancy)

Location is defined by (x,y,h) coordinates in a 2D orthogonal system where h is the heading angle

The North orientation of the X and Y axis is know.

Robot location reference is in the middle of the wheels axes.

The robot heading inside the orthogonal system is defined by h angle
The robot absolute heading is defined by No angle

Step 4: The "sonar" System

The system is made of 2 ultrasound sensors and a servomotor and is controlled by the Arduino Mega.

Ultrasonic sensors HY-SRF05 characteristics:

  • · Sound frequency 40kHz
  • · Measurement range 2cm to 450cm
  • · Measurement resolution 0.3cm
  • · Measurement angle 15°

Servomotor with metallic gears FT5316M

The system is used both for localization and obstacle detection.
For localization it makes scan by rotating the servomotor from 0 to 180° in 15 steps.

That means 13° rotation by step, a little bit less than the sensor measurement angle. 0° and 180° scans provide crossed front and back distances.

The 2 back to back HC-SRF05 sensors are alternatively triggered to provide a 360° scan.
Sensor returns 0 when it does not get any echo back to a trigger.

Step 5: The Global Software Architecture

A tensorflow Conv 2D model has been trained to learn each 30x30cm squares 360° echoes scan.

the points to bear in mind for this instrutable is that the robot transfers the raw data of scans to a "server application" (Java + Octave code) that run on a Windows server.

The "server application" converts the raw data to an image and provides it to the tensorflow model that returns a list of square locations with associated probabilities.

Step 6: What Are the Main Points to Address for Localizing With Echoes ?

Take into account the sensor limitations:

  • the complexity of the beam
  • the measurement range (400cm)

The complexity of echoes:

  • depends of the raw material of the obstacle
  • depends of the orientation of the obstacle
  • depends of the orientation of the robot
  • depends of the shape of the obstacles
  • depends of the distance of the obstacles

Step 7: An Example of Measurements Variability

In exactly the same condition, 2 consecutive scans provide both light and significative data differences.

But if we look at the data as an image they seems very close. That is why I thought images deep learning tools could be used as well for echoes recognition.

Step 8: How Moves Can Change Scans Image ?

This picture shows the impact of a move of around 30cm all around a central position (184,184) on the image of the 360 echoes scan.

We can see when the robot is geometrically translated, new image is not a simple geometric translation of dots

Step 9: The Challenges to Face

  1. be able to determine position when the robot is roughly at the learning conditions
  2. be able to approximate position when the robot is some between learning places
  3. be able to approximate position when robot heading précision is +-10°

I did some tries with simple logistic regression algorithm that were efficient to determine to face the challenge 1 only.

But with a Conv2D logistic regression algorithm it is possible to face all the challenges

Step 10: The Machine Learning Steps

Learning phase
Collect data
  • For each free position of the cartography (68) do 20 x 360° echoes scan, collect and store raw data with (x,y) location - with the robot 0° heading

Prepare data

  • define feature for each (x,y,0) learnt position
  • randomly split data between learning,validation and testing set

  • convert raw data echoes to B&W images and associate with a feature number

  • adjust data to Conv2D requirements

Train the model with learning and validation set

Testing phase

  • Test the determination with the testing set (robot is at the learning position)
  • Test the determination with the robot at new positions

Step 11: The Result

Learning needs few epochs. The point is to avoid over fitting.

I stopped learning after reaching 95% of accuracy with training and validation data.

I got also over 95% of accuracy with the testing data

So theoretically it works but what about real world ?

The first tests are encouraging.

  • Set the robot at a learned location but with a random heading
    • At first the robot is 0° oriented using compass. Due to the magnetic environment and compass limitation, the actual heading is +-8°.
    • Nevertheless the prediction is more than 90% the right one
  • Set the robot at a random location and with a random heading
    • The prediction is more than 90% the one of the closest possibility

Step 12: Determination Limitation

The above picture illustrates an example of determination result.

The robot has been set at a random position and then aligned to x axis with the compass.

There is a difference between theoretical and actual heading that will introduce noise in the measurements

  • The best possible prediction is (X1,Y1,0) the closest from the robot
  • The prediction is (X2,Y2,0)
  • The robot actual position is (X0,Y0,6)

So the predicted position is rather close to the actual one.

But to be able to run autonomously the determined location must be more precise. If (X,Y) error could be rather easily corrected by doing consecutive moves & scans, it is much more difficult to deal with the heading error that impatcs next (X,Y).

Step 13: How to Go Further ?

I am working on 2 different directions.

  • Firstly use particles filters as a mean to get location convergence with consecutive moves & scans.
  • Secondly run the prediction on a microcontroller (Portenta) with tensorflow lite instead of a windows system.

Step 14: Machine Learning Details

I use Colab to developp and train the tensorflow model.

To reduce matrix size echo distances are divided by 10. That means the maximal X and Y values are 80 and the minimal are -80.

1 pixel = a 10x10cm square - above an example of reduced size image

Prediction code is written in python without Colab

The Model

_________________________________________________________________
Layer (type) Output Shape Param # =================================================================

conv2d_8 (Conv2D) (None, 80, 80, 8) 16 _________________________________________________________________

max_pooling2d_8 (MaxPooling2) (None, 40, 40, 8) 0 _________________________________________________________________

conv2d_9 (Conv2D) (None, 40, 40, 16) 1168 _________________________________________________________________

max_pooling2d_9 (MaxPooling2) (None, 20, 20, 16) 0 _________________________________________________________________

flatten_4 (Flatten) (None, 6400) 0 _________________________________________________________________

dense_4 (Dense) (None, 68) 435268 =================================================================