Introduction: Smart Robot With Vision and Voice Control

How to make an intelligent robot with computer vision, voice control, voice synthesis and more, using your Android phone!

Step 1: Parts


1. An Android phone that supports OTG with Android KitKat or later installed. Most modern phones support OTG. (I used a UMI Emax here cos it's powerful and cheap!)

2. An Arduino Uno based robot platform. I recommend the MakeBlock Starter Robot Kit (you can use either IR or Bluetooth version).

3. The top part of a dashboard mobile phone holder (I used a BlackFox Digidock CR-3104)

4. An OTG cable (easily found on ebay or Amazon)

5. A standard micro USB cable (preferably a very short one, or adapter module if you can find one)

6. A magnetic periscope lens (easily found on Amazon or ebay)


1. Arduino IDE

2. DroidScript IDE

Step 2: Make the Robot Base

1. First build the standard tracked robot model according to the MakeBlock instructions.

2. Check that you can drive the robot using the included infra-red controller.

Step 3: Reconfigure the Hardware

1. Remove the battery box and strap it below the robot using two of the cable ties.

2. Take off the ultra sonic sensor and it's metal plate, then re-attach them in the lower position shown.

3. Attach the ultra-sonic range sensor to port 4 on the controller board.

Step 4: Attach the Phone Holder

- Attach the phone holder to the top of your robot using cable ties, double sided tape or Velcro pads.

- Place your phone in the holder and attach the periscope lens to your phone's camera.

- Connect the OTG cable and micro USB cable together and then connect them between your phone and the robot's control board. Make sure that the OTG cable goes into your phone and not the other way around or it won't work.

Step 5: Program the MakeBlock Orion (Arduino Uno) Board

- Install the Arduino IDE

- Install the MakeBlock Arduino libraries:

- Connect the robot controller board to your computer using a standard micro USB cable.

- Select "Arduino Uno" as the board type from the 'Tools' menu.

- Create the following new sketch (i.e. program):-

// Smart Rover - Arduino Sketch for controlling a // Makeblock robots from DroidScript via OTG cable. // // This program can be tested using the Aurduino serial // monitor. // // Examples: // type 'lft100x' to turn left at speed 100. // type 'buzy' to turn buzzer on, 'buzn' to turn it off. // // Copyright: // License: Creative Commons Attribution ShareAlike 3.0 //--------------------------------------------------------

#include <MeOrion.h>
#include <Arduino.h> #include <SoftwareSerial.h> #include <Wire.h>

//Global variables. char g_version[] = "0.10\n"; MeDCMotor g_motorL( M1 ); MeDCMotor g_motorR( M2 ); MeUltrasonicSensor ultraSensor( PORT_4 ); unsigned long timer = 0;

//Setup the hardware. void setup() { //Setup USB serial comms. Serial.begin( 115200 ); Serial.setTimeout( 100 ); }

//This function is called forever. void loop() { //Read serial commands. while( Serial.peek() != -1 ) { //Read 3 character command. char cmd[4] = "---"; Serial.readBytes( cmd, 3 ); //Execute command. if( strcmp( cmd, "buz" )==0 ) Buzzer(); else if( strcmp( cmd, "stp" )==0 ) Stop(); else if( strcmp( cmd, "fwd" )==0 ) Forward(); else if( strcmp( cmd, "rev" )==0 ) Reverse(); else if( strcmp( cmd, "lft" )==0 ) Left(); else if( strcmp( cmd, "rgt" )==0 ) Right(); else if( strcmp( cmd, "ver" )==0 ) GetVersion(); } //Report status every second. if( (millis()-timer) > 1000 ) { timer += 1000; Report(); } }

//Get software version. void GetVersion() { Serial.print( g_version ); }

//Send status back to DroidScript App. void Report() { int dist = ultraSensor.distanceCm(); Serial.println( String("dist:") + dist + String(";") ); }

//Control buzzer. void Buzzer() { //Read on/off parameter. char onOff; Serial.readBytes( &onOff, 1 ); //Start of stop buzzer. if( onOff=='y' ) buzzerOn(); else buzzerOff(); }

//Turn vehicle left. void Left() { //Read speed parameter. int speed = Serial.parseInt(); //Turn on motors. -speed ); speed ); }

//Turn vehicle right. void Right() { //Read speed parameter. int speed = Serial.parseInt(); //Turn on motors. speed ); -speed ); }

//Drive vehicle forward. void Forward() { //Read speed parameter. int speed = Serial.parseInt(); //Turn on motors. speed ); speed ); }

//Reverse vehicle. void Reverse() { //Read speed parameter. int speed = Serial.parseInt(); //Turn on motors. -speed ); -speed ); }

//Stop vehicle. void Stop() { //Turn off motors. 0 ); 0 ); }

Step 6: Create the Phone App

- Install the free DroidScript IDE App from Google Play.

- Navigate to the following link using your phone's browser and download the DroidScript project (.SPK) file from here:-

- Use a file browser such as Explorer+ or ES File Explorer and open the downloaded .spk file. The .project file will then be automatically installed into DroidScript (Select 'Yes' when it asks you if you trust the source).

Step 7: Take It for a Spin!

- Place batteries in the controller board and turn on the sliding power switch.

- Insert (or re-insert) the OTG cable into your phone (select DroidScript as the default USB program if prompted)

- Start the 'Rover' App by touching it's icon in DroidScript.

- Press the "fwd" key and then the "3" key, then press "Run" button (your rover should drive foward for 3 seconds).

- Press the "Voice" key and try speaking to it (Note: its often best to download offline voice recognition in the Android Settings if you don't have a fast internet connection).

- Try putting your hand in front of the ultrasonic sensors while it is moving foward.

- Try holding colored objects in front of the camera.

Step 8: Further Experiments

There are loads more cool things that can be added to this robot using DroidScript, such as GPS navigation, compass orientation, face tracking,motion detection, remote camera streaming etc. Just look through the DroidScript samples and see what you can find.

The best way to program your robot is to use DroidScript's WiFi IDE. This lets you connect to your phone via WiFi and remotely edit your programs using using a web browser running on your PC or Mac (just press the little arrow button at the top left on the DroidScript App to connect).

Have fun!


Dave-Smart (author)2017-07-31

Sounds like you have one of your motor wires swapped. Try swapping over the wires on one of your motors.

Got you monkey (author)2017-07-30

There is a problem. When I click left the bot goes forward and when I click right it goes backward. When I click forward it goes left and when I click backward it goes right. What might be the problem. Please help!

MattK120 (author)2016-09-27

Amazing tutorial, thank you!

My kids want their MBot to talk, I have extra Android phones lying around the house, and this should do the trick!

Dave-Smart (author)2016-08-22

Color detection, motion detection and face detection are all included internally within DroidScript and run locally (it's amazing how powerful phones are these days). We are also working on an OpenCV vision plugin which will be released as open source very soon :)

EdJ2 (author)2016-08-21

Great post!

Did you perform the object color detection locally as opposed to streaming the data to a more powerful box? Either way, very curious how you handled that computer vision functionality.


LeeM91 (author)2016-07-02

Thanks Dave

I see that you have managed to update the code sample. However it shows a reference to MeOrion.h rather than Makeblock.h. I assume MeOrion is correct, as I get an error when using Makeblock. It loaded fine after the changes were made and we were able to get the voice control working.

For the standard wiring of the motors I think that."forward" will turn the robot the code is running both motors at positive speed. One motor is actually reversed in the chassis, so forward actually involves running one motor at negative speed. But this will depend on how anyone has wired up the motors. Just switch the wires of one motor over if you find it is turning on the forward command.

Dave-Smart (author)LeeM912016-07-03

Yes, MakeBlock have updated their Arduino libs since I wrote this article, so you need to use #include <MeOrion.h> instead now.

I'm glad you got it working, if you fancy trying to stream video back to your PC to make a remotely controlled rover, then you can use the built-in "Camera Stream" sample in DroidScript or you could stream to another Android device using this method:-

LeeM91 (author)Dave-Smart2016-07-03

I found it difficult to use the voice control though. Possibly because there was too much background noise. I have all these exciting ideas for an autonomous robot with voice control and I really want that part working well. Passing commands to google and back would be really useful if you can feature a solution about that. Getting the latest weather forecast for example.

LeeM91 (author)2016-07-01

Thanks for this great idea. However the code sample you provided does not compile.

Am I correct in understanding that the 4 include statements without a filename should be replaced with the include statements relevant to your specific board? So with the starter robot you used, the include statements would be as below. Is that correct?

#include <Arduino.h>
#include <Wire.h>
#include <SoftwareSerial.h>
#include <MeOrion.h>

I have made this change, but when I try to upload the code I get an error stating "avrdude stk500_cmd() programmer is out of sync"

Do you know what might be the issue?

dushyantahuja (author)2015-09-08

Really cool - had not heard about droid-script before.

tomatoskins (author)2015-09-08

This is great! Thanks for sharing!

About This Instructable




More by Dave-Smart:Smart Robot with Vision and Voice control
Add instructable to: