Introduction: Alexa Voice Controlled Raspberry Pi Drone With IoT and AWS
Hi! My name is Armaan. I’m a 13-year-boy from Massachusetts. This tutorial shows, as you can infer from the title, how to build a Raspberry Pi Drone. This prototype demonstrates how drones are evolving and also how big a part they might play in the future. I can definitely see myself waking up in 10 years and asking a drone to get breakfast for me. The drone uses Amazon Alexa, Amazon Web Services, IoT(Internet of Things), and most importantly a Raspberry Pi to run. It is meant to demonstrate and inform about drones and how they are improving every day. Hopefully you are successful and learn about drones in the process. Good luck and thanks for reading. -Armaan
Supplies
To build the prototype there are various hardware and software needs. I used an online tutorial by The Drone Dojo to build the drone and integrated the technologies listed.
For the drone you can find the parts list right here:
Software Requirements:
- Amazon Web Services
- A Laptop
- Mission Planer Software
- Balena Etcher
- MicroSD Card with Raspbian File found here
- Amazon Alexa, physical or virtual
Step 1: Gathering and Understanding Parts
Every part mentioned in the supply list is necessary, and so is a clear understanding of every part. You can find the parts online and once gathered, continue reading. A playlist by The Drone Dojo for a full understanding of the parts can be found here. A 4-minute explanation on my youtube channel can be found here. When it comes to drones the only parts, despite what most people think, are not just motors and propellers. Below are the basic purposes of each part.
- The Raspberry Pi with the Emlid Navio2
This part is pretty much a central processor and the main point of the drone. The Raspberry Pi acts like the CPU of a computer which sends commands to the Navio2 to execute through PWM(Pulse Width Modulation Signals) to other parts of the drone
2. The ESC's(Electronic Speed Controllers)
These yellow parts are found beneath the frame. They are 4 plugged in to the Navio, one for each motor. Upon recieving PWM signals, they rotate the motors and begin flight.
3. Motors
Motors don't need too much explanation because you are probably familiar with them. They rotate and spin the propellers to create thrust.
4. Propellers
Propellers create thrust for the drone to fly. They spin in the same direction as the motors to lift the vehicle.
5. The Battery and Power Module
The LiPo Battery powers the whole drone through the frame using the power module. It gives about 15-20 minutes of flight time and acts as a power source.
6. The GPS
The GPS communicates with satellites to determine the drone's position. It determines altitude, latitude, and longitude. It can be used for Geofencing, waypoints, and also moving to certain positions or directions.
7. The Telemetry Module
The telemetry module connects our drone to a ground control station, in our case Mission Planner, to be monitored.
8. The RC Controller and Module along with PPM Encoder
The RC Controller uses radio to transmit signals and commands to the RC Module to pilot the drone manually. The PPM Encoder translates these signals for the Navio + RPI to process and execute.
9. The Frame
This red and white frame acts as a base or platform for the others parts to be placed. The frame is aerodynamic and lightweight, therefore perfect for our drone build.
Now with knowledge of every part, we can finally build the drone! Onwards to the next step!
Step 2: Assembling the Drone
This step is probably the most difficult in terms of physical effort. I recommend having another person for help or try to use the helping hand in the parts list. The process is too long to demonstrate here, so I will provide another link I used by The Drone Dojo.
Again, as I won't go into too much detail, I'll just highlight the basics of each step.
1. Organize Your Supplies - Gather our materials and make sure they're easily accessible
2. Plan Your Build - Organize your parts on the frame to make a blueprint of what you'll build
3. Soldering Work - This is the work that's a little hard to do alone. You must solder the golden bullet connectors that come with the motors to the ESC's. Next, you must take the bottom part of the frame and solder the ESC's to the bottom frame or Power Distribution Board. The Battery Module will also be soldered on to the Power Distribution Board
4. Setting Up The Frame - You must then screw the top part of the frame in along with the arms. You can then attach the Raspberry Pi on top any way you want.(I used duct tape). Then you can secure the ESC's to the arms with zip-ties. Now we're almost done.
5. Binding the RC Controller to Receiver - Try following the instructions in the playlist above to bind the RC Controller using an ESC.
6. Finalizing Parts on Frame - Duct tape or strap on the telemetry module on to the frame. Duct tape the PPM Encoder to an arm as well. Now you can wire the ESC's and PPM Encoder to the Navio.
7. GPS Mount + Battery - Assemble the GPS Mount with the various screws and pieces. Using zip-ties, attach the GPS to the Frame. I didn't necessarily use the GPS Mount because of its fragility, but it's up to you. Next, you can insert the battery between the Power Dist. Board. I strapped and taped on the power module to the frame as well. Now your hardware is pretty much set up. Now for the part we've waiting for!
8. Installing the Propellers!!! - You can tighten the propellers using the chart mentioned in the playlist. Then you can plug in the ESC's to the motors and we're finally finished building the drone.
Software's next, so onwards!
Step 3: Configuring Raspberry Pi and GCS(Mission Planner)
Again, you can find more detailed instructions in the playlist from the last step. However, you probably know how to setup the RasPi. But this time, we're doing it headless. Use Balena Etcher to burn the OS from the Navio OS website into the MicroSD card. While its plugged in to your computer, go into the wpa supplicant using notepad++. After that, enter the ssid and password for the Raspberry Pi to connect to your WiFi. Then you must add a file called SSH. This can be through the Command Line or another method. Now we can SSH. You can use command prompt or Putty. I used command prompt and typed "ssh pi@navio" to connect in my case, or you can find the IP address and ssh that way. Upon connecting, use this video to set up and configure the Navio. To setup the telemetry you must first make an edit on the Raspberry Pi. Follow this to make the edit and try to connect to Mission Planner. If telemetry doesn't work, you can undo the edit and connect using a UDB connection by entering your GCS(Ground Control Station such as laptop) IP. Once connected to Mission Planner, you can use the setup wizard to calibrate all parts of the drone. If you need help refer again to the playlist. Usually, whenever you set up, there's almost always an error. Troubleshooting is one of the biggest parts of this project. I can't really help you there as I am not aware of your errors, but most errors can be fixed with help from the internet. After everything is ready, then the drone is ready to fly! You can setup your RC controller and flight modes on Mission Planner. Try to hold the left stick to the very bottom-right for five seconds to arm the drone. I don't recommend flying without looking at a tutorial because the drone is very fragile and easy to break. For me, the first time I flied it, I broke the GPS Mount and some propellers. If you don't require voice control, then you can stop here. To learn about AWS and programming the drone continue!
Step 4: Programming the Drone to Fly With Python
Before getting in to AWS, we should first understand how to program the drone to fly. After the initial setup, integrating voice control shouldn't be too difficult. The first thing we can try to get a sense of it is making a simple takeoff and land program. After setting up the Raspberry Pi, we can SSH into it again. You can look at the playlist again or follow my instruction.
1. First let's download the ArduPilot Source Code in a directory on the Raspberry Pi
mkdir src
Then, get the files from GitHub using git clone
git clone -b Copter-3.6.11 https://github.com/ardupilot/ardupilot
Now, let's navigate to /src/ardupilot
cd src cd ardupilot
Now, let's initialize the source files
git submodule update --init --recursive
2. Next, we have to compile the firmware on the Raspberry Pi
Make sure to navigate to /src/ardupilot/ with cd before doing next steps
Then to cofigure specifically to the Navio using
./waf configure --board=navio2
Then you can compile with
./waf --targets bin/arducopter
3. We can now install the source code to the Navio
First lets navigate to the right directory.
cd/etc/systemd/system
Then edit the file
sudo vi arducopter.service
Where it says ExecStart, insert the following instead of what's already there
ExecStart=/bin/sh -c "/home/pi/src/arducopter/build/navio2/bin/arducopter ${ARDUPILOT_OPTS}"Now, to put the ardupilot source code into action, we can use
sudo systemctl daemon-reload
Then we can restart with
sudo systemctl restart arducopter
With the last step, we have finally finished setting up ArduPilot on our drone
4. Installing DroneKit
DroneKit is the software that we will use to program the drone to fly. To understand some of the code you can find the documentation here. First we must install the package on our drone before writing a script.
We can install the python package with
pip install dronekit==2.9.1
This may, or may not be newest version, but it is the one I used so I can help with troubleshooting.
To check if installed, we can do a
pip freeze | grep dronekit
Now we're finally ready to make our first python script
5. takeoff_and_land.py
WARNING! I suggest establishing a basic understanding of python, so you can learn and understand the code. If you want to write the program yourself follow this video.
##First let't create a directory to store this code cd dk ##If you want to do it yourself then use vi takeoff_and_land.py ##to create a program
Otherwise, you can take a look or use the file attached and use a file transfer protocol. We can try out this program afterwards. First to clarify that it is a python file we must use
chmod +x takeoff_and_land.py
Then, to try use the following code to run
python takeoff_and_land.py --connect 127.0.0.1:14550
The first time it did not work for me either. If there is a link timeout, then don't worry there is something you can do. Open another prompt and ssh. You can try installing something called mavproxy and try running it. After that, you can run both simultaneously. This should help connect the drone. Once that is done, I have a challenge for you. Try to figure out what the other program(set_velocity_body.py) does and how to make it work. If you do, good job.
6. Onward!
We can now use this knowledge to make our drone voice-controlled. Alexa drone control uses a lot of these features and more. Good luck and onward!
Step 5: Using Amazon Alexa and Amazon Web Services to Integrate Voice Control
This step is one of the less documented ones. This means that it will be the hardest to troubleshoot. Just getting it to work took me about a month, maybe more. The most important thing here is to be patient. This feature, if implemented in real life, can be life-changing. You can just tell Alexa to get your groceries for you instead of going yourself. Imagine that! So without further ado let's get into it!
1. Registering the Raspberry Pi as a Thing on AWS IoT
To use IoT(Internet of Things), well we need a thing. So we have to login to the AWS Console to use AWS IoT first. Then go to IoT Core. Once there, you should click Manage and then create a thing. After adding a name, for connecting, we need a certificate. I would recommend clicking the One-click certification. Then after seeing the certificate screen, make sure to download each and every key including the root CA. Then you can go and finish up the creation of the thing. Next we have to create a policy. Go back out to the IoT Core. Then click on secure and click policies. Then hit create policy. You can then create a name and add resources. Under action, type iot* and type * under resource and hit allow for effect. Then go back to your thing and go to your certificate. Once here, click on policies. You can then attach your policy for the thing and it's all set!
2. Setting up Code on Raspberry Pi and interacting with IoT
For this part, you are going to need a SFTP Client(I used WinSCP) for file transfer. Upon connecting to our Raspberry Pi, we will need to have the certificate keys on hand. You must transfer the key files to the Raspberry Pi. You should also pip install the AWSIoTPythonSDK on the Raspberry Pi. Then go in to the dk directory on the Raspberry Pi. You use the Alexa Drone Control file I gave for communicating with IoT. To use this file I used a Shell Script for starting up. I will show the code below as I can't upload the file for some reason. If the program doesn't pick up messages from AWS IoT while testing, don't worry! This might be my fault as the Alexa Drone Control File may not suit your Thing. So, to fix it, go back to AWS IoT and hit learn on the left panel. Follow the instructions and you may have to restart. Sorry about that. Once your IoT starts working with the program on the RasPi, you can integrate the dronekit code from the Alexa Drone Control file I gave. After that, use the Shell Script I gave with your certificates and Rest API Endpoint from IoT.
<p># stop script on error<br>set -e # Check to see if root CA file exists, download if not if [ ! -f ./root-CA.crt ]; then printf "\nDownloading AWS IoT Root CA certificate from AWS...\n" curl <a href="https://www.amazontrust.com/repository/AmazonRootCA1.pem" rel="nofollow"> https://www.amazontrust.com/repository/AmazonRoot...> > root-CA.crt fi # install AWS Device SDK for Python if not already installed if [ ! -d ./aws-iot-device-sdk-python ]; then printf "\nInstalling AWS SDK...\n" git clone <a href="https://github.com/aws/aws-iot-device-sdk-python.git" rel="nofollow"> https://www.amazontrust.com/repository/AmazonRoot...> pushd aws-iot-device-sdk-python python setup.py install popd fi</p><p># run pub/sub sample app using certificates downloaded in package printf "\nRunning pub/sub sample application...\n" python dk/AlexaDroneControl.py --connect 127.0.0.1:14550 -e "Your IoT ARN" -r root-CA.crt -c PiGroundStation01.cert.pem -k PiGroundStation01.private.key</p><br>
This won't work for all of you as the names are different. Instead of the keys that I used replace with your key names when you transfer the file. Make sure to exit dk before transferring the file! That should be all you have to do for now.
3. Building your Alexa Skill
This step seems a lot harder than it really is. First, we must log into the Alexa Developer console. Then, just hit create skill. When it asks to choose a model for your skill, just hit custom. When it asks to choose a method, hit provision your own. You can name it whatever you want. No template is required, so choose start from scratch. Next, after creating your skill, you should arrive on the Skill builder screen with a checklist on the right. From here, we can start building our skill. First on the checklist is the invocation name. This is what you will tell Alexa to invoke your skill. For me, I just put my invocation name as drone. Now we can go to our intents, utterances, and slots. Here, you can make commands for the drone such as go up 1 meter or go right or down. Mine only really works with one meter for now. You can click the JSON Editor on the bottom of the left panel. You can then paste the following code in to it.
<p>{<br> "interactionModel": {
"languageModel": {
"invocationName": "drone",
"intents": [
{
"name": "AMAZON.FallbackIntent",
"samples": []
},
{
"name": "AMAZON.CancelIntent",
"samples": []
},
{
"name": "AMAZON.HelpIntent",
"samples": []
},
{
"name": "AMAZON.StopIntent",
"samples": []
},
{
"name": "AMAZON.NavigateHomeIntent",
"samples": []
},
{
"name": "GoIntent",
"slots": [
{
"name": "Direction",
"type": "Direction"
},
{
"name": "Distance",
"type": "AMAZON.NUMBER"
},
{
"name": "Unit",
"type": "Unit"
}
],
"samples": [
"Go {Direction} {Distance} {Unit}",
"Go {Distance} {Unit} {Direction}"
]
},
{
"name": "CommandIntent",
"slots": [
{
"name": "Task",
"type": "Task"
}
],
"samples": [
"Command drone to {Task}"
]
},
{
"name": "TurnIntent",
"slots": [
{
"name": "Direction",
"type": "Direction"
},
{
"name": "Rotation",
"type": "AMAZON.NUMBER"
}
],
"samples": [
"Turn {Direction} {Rotation} degrees"
]
}
],
"types": [
{
"name": "Direction",
"values": [
{
"name": {
"value": "Straight",
"synonyms": [
"Forwards",
"Forward"
]
}
},
{
"name": {
"value": "Back",
"synonyms": [
"Backwards",
"Backward"
]
}
},
{
"name": {
"value": "Right"
}
},
{
"name": {
"value": "Left"
}
},
{
"name": {
"value": "Down"
}
},
{
"name": {
"value": "Up"
}
}
]
},
{
"name": "Unit",
"values": [
{
"name": {
"value": "Meters",
"synonyms": [
"Meter"
]
}
},
{
"name": {
"value": "Yards",
"synonyms": [
"Yard"
]
}
},
{
"name": {
"value": "Feet",
"synonyms": [
"Foot"
]
}
}
]
},
{
"name": "Task",
"values": [
{
"name": {
"value": "Land"
}
},
{
"name": {
"value": "Hold"
}
},
{
"name": {
"value": "Stay"
}
},
{
"name": {
"value": "R.T.L.",
"synonyms": [
"Return to Launch"
]
}
}
]
}
]
}
}
}</p><br>After pasting the code in to the JSON Editor, you can click on the third step of the checklist and it will build your interaction model for you. With this step, you will be done for now. You can leave the endpoint slot blank for now!
Step 4: Building Your Lambda Function
Now, this step is one that you are going to have to figure out for yourself. I will tell you how to connect it to the Alexa Skill, but you'll have to code it yourself. So, first go to AWS management console. Then, go to Lambda. You can then create a function naming it whatever you want. Make sure to author it from scratch and make the runtime whatever programming language you want. I used Node.js. To add the Alexa Skill, add a trigger and select Alexa Skills Kit(ASK). Copy your Lambda ARN and go back to the Alexa Skill. Now, go to endpoint. You can paste your ARN, save, and build a new model. Then, take the Alexa Skill ID, and paste it in the trigger section where it asks you on the Lambda. Next, scroll down on the Lambda and find the basic settings and make the timeout 10 seconds. Now, it's up to you to figure out the code. For hints, you can the websites below.
https://github.com/aws/aws-iot-device-sdk-js
https://www.hackster.io/veggiebenz/voice-controlle...
and you can use the file I attached, but it is incomplete and will not work.
/**
* Control your APM / Pixhawk quadcopter with your voice, using Amazon Alexa, Lambda, 2lemetry MQTT. */ var awsIot = require('aws-iot-device-sdk'); var config = require("./config"); var deviceName = "EchoDroneControl"; // this device is really the controller var mqtt_config = { "keyPath": config.privateKey, "certPath": config.certificate, "caPath": config.rootCA, "host": config.host, "port": 8883, "clientId": "Lambda-" + deviceName, //+ "-Lambda-" + (new Date().getTime()), "region":"us-east-1", "debug":true }; var ctx = null; var client = null; // Route the incoming request based on type (LaunchRequest, IntentRequest, etc.) The JSON body of the request is provided in the event parameter. exports.handler = function (event, context) { try { console.log("event.session.application.applicationId=" + event.session.application.applicationId); ctx = context; if (event.session.application.applicationId !== app_id) { ctx.fail("Invalid Application ID"); } client = awsIot.device(mqtt_config); client.on("connect",function(){ console.log("Connected to AWS IoT"); // callback(); });if (event.session.new) { onSessionStarted({requestId: event.request.requestId}, event.session); } if (event.request.type === "LaunchRequest") { onLaunch(event.request, event.session); } else if (event.request.type === "IntentRequest") { onIntent(event.request, event.session); } else if (event.request.type === "SessionEndedRequest") { onSessionEnded(event.request, event.session); ctx.succeed(); } } catch (e) { console.log("EXCEPTION in handler: " + e); ctx.fail("Exception: " + e); } }; /** * Called when the session starts. */ function onSessionStarted(sessionStartedRequest, session) { console.log("onSessionStarted requestId=" + sessionStartedRequest.requestId + ", sessionId=" + session.sessionId); }
/** * Called when the user launches the skill without specifying what they want. */ function onLaunch(launchRequest, session, callback) { console.log("onLaunch requestId=" + launchRequest.requestId + ", sessionId=" + session.sessionId); // Dispatch to your skill's launch. getWelcomeResponse(callback); } /** * Called when the user specifies an intent for this skill. */ function onIntent(intentRequest, session ) { //, callback) { console.log("onIntent requestId=" + intentRequest.requestId + ", sessionId=" + session.sessionId); var intent = intentRequest.intent, intentName = intentRequest.intent.name; console.log("REQUEST to string =" + JSON.stringify(intentRequest)); var callback = null; // Dispatch to your skill's intent handlers if ("GoIntent" === intentName) { doGoIntent(intent, session); } else if ("CommandIntent" === intentName) { doCommandIntent(intent, session); } else if ("TurnIntent" === intentName) { doTurnIntent(intent, session); } else if ("HelpIntent" === intentName) { getWelcomeResponse(); } else { throw "Invalid intent"; } } /** * Called when the user ends the session. * Is not called when the skill returns shouldEndSession=true. */ function onSessionEnded(sessionEndedRequest, session) { console.log("onSessionEnded requestId=" + sessionEndedRequest.requestId + ", sessionId=" + session.sessionId); // Add cleanup logic here } // --------------- Functions that control the skill's behavior ----------------------- function getWelcomeResponse() { // If we wanted to initialize the session to have some attributes we could add those here. var sessionAttributes = {}; var cardTitle = "Welcome"; var speechOutput = "Welcome to the DRONE CONTROL . "; // TODO: is drone online or offline? If online, is it ARMED? var repromptText = "Drone ready for command."; var shouldEndSession = false; ctx.succeed(buildResponse(sessionAttributes, buildSpeechletResponse(cardTitle, speechOutput, repromptText, shouldEndSession))); } /** * handles GO intent. */ function doGoIntent(intent, session, callback) { // var cardTitle = "Drone GO..."; var repromptText = ""; var sessionAttributes = {}; var shouldEndSession = false; var speechOutput = ""; var direction = intent.slots.Direction.value; var distance = intent.slots.Distance.value; var unit = intent.slots.Unit.value; var validDirections = ["forward", "forwards", "backwards", "back", "right", "left", "up", "down", "straight", "ahead", "straight ahead"]; var validUnits = [ "foot", "feet", "meter", "meters", "yard", "yards" ]; repromptText = "Tell me how far to go and in what direction. "; var fail = false; // validate inputs if ( !( parseInt(distance) >= 1 ) ) { speechOutput = "I couldn't understand the distance you want me to travel. "; fail = true; ctx.succeed(buildResponse(sessionAttributes, buildSpeechletResponse(cardTitle, speechOutput, repromptText, shouldEndSession))); } if (validDirections.indexOf(direction) == -1) { speechOutput = "I couldn't understand the direction you want me to travel. "; fail = true; ctx.succeed(buildResponse(sessionAttributes, buildSpeechletResponse(cardTitle, speechOutput, repromptText, shouldEndSession))); } if (validUnits.indexOf(unit) == -1) { speechOutput = "I couldn't understand the unit you want me to travel. "; fail = true; ctx.succeed(buildResponse(sessionAttributes, buildSpeechletResponse(cardTitle, speechOutput, repromptText, shouldEndSession))); } if (!fail) { var cardTitle = "Drone going " + direction + " " + distance + " " + unit; speechOutput = "Going " + direction + " " + distance + " " + unit; mqttPublish(intent, sessionAttributes, cardTitle, speechOutput, repromptText, shouldEndSession); } }
function doCommandIntent(intent, session, callback) { // var cardTitle = "Drone COMMAND..." ; var repromptText = null; var sessionAttributes = {}; var shouldEndSession = false; var speechOutput = ""; repromptText = "Tell me what is the command for the drone. "; var task = intent.slots.Task.value; var validTasks = [ "launch", "land", "r. t. l.", "hold", "stay", "stop", "return to launch", "abort" ]; if (validTasks.indexOf(task) == -1) { speechOutput = "I couldn't understand the command. "; ctx.succeed(buildResponse(sessionAttributes, buildSpeechletResponse(cardTitle, speechOutput, repromptText, shouldEndSession))); } else { var cardTitle = "Executing Drone command " + task ; speechOutput = "Executing command " + task; mqttPublish(intent, sessionAttributes, cardTitle, speechOutput, repromptText, shouldEndSession); } }
function doTurnIntent(intent, session, callback) { // var cardTitle = "Drone Turn..." ; var repromptText = null; var sessionAttributes = {}; var shouldEndSession = false; var speechOutput = ""; repromptText = "Tell me how you want to turn the drone. "; var direction = intent.slots.Direction.value; var validDirections = [ "right", "left", "around" ]; if (validDirections.indexOf(direction) == -1) { speechOutput = "I couldn't understand the direction of the turn. "; ctx.succeed(buildResponse(sessionAttributes, buildSpeechletResponse(cardTitle, speechOutput, repromptText, shouldEndSession))); } else { var cardTitle = "Drone turning " + direction; speechOutput = "Turning " + direction; mqttPublish(intent, sessionAttributes, cardTitle, speechOutput, repromptText, shouldEndSession); } }
function mqttPublish(intent, sessionAttributes, cardTitle, speechOutput, repromptText, shouldEndSession) { var strIntent = JSON.stringify(intent); console.log("mqttPublish: INTENT text = " + strIntent); // client.publish("ikw1zr46p50f81z/drone/echo", strIntent, false); client.publish(config.topic, strIntent, false); client.end(); client.on("close", (function () { console.log("MQTT CLIENT CLOSE - thinks it's done, successfully. "); ctx.succeed(buildResponse(sessionAttributes, buildSpeechletResponse(cardTitle, speechOutput, repromptText, shouldEndSession))); })); client.on("error", (function (err, granted) { console.log("MQTT CLIENT ERROR!! " + err); })); }
// --------------- Helpers that build all of the responses ----------------------- function buildSpeechletResponse(title, output, repromptText, shouldEndSession) { return { outputSpeech: { type: "PlainText", text: output }, card: { type: "Simple", title: title, content: output }, reprompt: { outputSpeech: { type: "PlainText", text: repromptText } }, shouldEndSession: shouldEndSession } } function buildResponse(sessionAttributes, speechletResponse) { return { version: "1.0", sessionAttributes: sessionAttributes, response: speechletResponse } }
Attachments
Step 6: Have Fun and Test It!
Participated in the
Raspberry Pi Contest 2020
