Introduction: Flight Simulator With Arduino and Python

Picture of Flight Simulator With Arduino and Python
Control the Flight Simulator option of Google Earth using an Arduino and Accelerometer.

Fly virtually around the world; tilting the Accelerometer forward, back, left and right to control the Plane's Pitching and Banking.

Objectives:
* Learn how to send Serial data from Arduino to Python
* Learn how to manipulate strings and lists in Python to read and interpret the data sent from Arduino
* Learn how to use the pywin library for Python to control Windows functions such as Mouse and Keyboard


We use Python as the interface between Arduino and Google Earth. In my Arduino / Raspberry Pi Internet Radio Instructable (https://www.instructables.com/id/Arduino-Raspberry-Pi-Internet-Radio/) we used nanpy on both Python and Arduino so that Python could issue commands to Arduino and read responses. Life would be boring if we always did stuff the same way, so in this project Arduino talks over Serial USB to Python by use of the pyserial library for Python.


If you don't have Google Earth on your PC download it from here:
http://www.google.com/earth/index.html

BTW, you if you don't want to agree to Google Chrome becoming your Default Browser UNCHECK the box at the License Agreement stage.

Step 1: You Will Need

Picture of You Will Need

You will need:

* A Windows PC
* Google Earth installed on the PC
* Python 3.2 installed on the PC
* An Arduino Uno
* An Accelerometer (e.g. http://goo.gl/ZddGy )
* A prototyping system or breadboard to wire the Accelerometer to the Arduino (e.g. http://goo.gl/1E1iI)
* A small push switch

Step 2: Install the Required Python Libraries

If you don't have Python already installed, get it from here:
http://www.python.org/download/releases/3.2.3/

I recommend Python 3.2 (as opposed to Python 2.7) because it's the supported version going into the future and many of the popular libraries such as pyGame, nanpy, pyserial, have been ported to 3.2. I also recommend the 32bit version for increased compatibility with libraries (even on a 64bit machine).

1.
We'll be using pyserial to communicate with the Arduino. There's an excellent guide to installation here:
http://goo.gl/L5o78

and you can find the library here:
http://pypi.python.org/pypi/pyserial

2.
Python will control the flight simulator by emulating the PC's mouse. To do that we need the pywin32 library for Python.

Take a look at how to install it here:
http://sourceforge.net/projects/pywin32/files/

I used the win32 installer for python3.2 (32bit)  from here:
http://goo.gl/lO0DB



Step 3: Wire the Accelerometer and Switch to the Arduino

Picture of Wire the Accelerometer and Switch to the Arduino

Wire the accelerometer to the Arduino as shown in the Diagrams below:

* 3V => 3.3V
* GND => GND
* X => A0
* Y => A1
* Z => A2

BEWARE:  The Power to the ADXL335 is 3.3V.  Connecting it to 5V will KILL it.

Add a Push Switch between Digital Pin0 and GND

Step 4: Arduino Tilt Control

Picture of Arduino Tilt Control

There are several components to this project:

1. Tilting the Arduino Accelerometer sends data to Python on the PC
2. Python moves the PC's Mouse
3. Google Earth runs the Flight Simulator according to the Mouse Position

We'll start with getting the Arduino to sense the Tilt of the Accelerometer and send the data over the serial port (over USB) to the PC.

The full code is shown in the Next Step. For now let's see the main ideas that make the code work.

1.
At the start of the code we need to establish what readings from the Accelerometer correspond to level. We need to do this so we'll know when the device has been tilted away from level.

We write a function called average() using a for loop to measure the relevant data from the pin and average it as follows:

// Get the Average of 8 readings from 'pin'
int average(int pin) {
  int Ave = 0;
  for (int i=0; i<8; i++) {
    Ave = Ave + analogRead(pin);
  }
  return Ave/8;
}


We call average() for the x, y and z axis measurements during the setup() section that is run once, at the beginning of the program.

2.
We establish the device is tilted by a simple subtraction as follows:
xTilt = analogRead(xpin)-xAve;

3.
We need to communicate the State of the Pushbutton Switch to Python so it can takeover mouse control when pressed and relinquish it when pressed again.

To do this we need to enable a Pullup on  swpin in the setup() section:
digitalWrite(swpin, HIGH);    // Enable Pullup on Switch Pin

and read the pin in the loop() section:
swState = digitalRead(swpin);        // swState = 0 when Pressed


4.
We communicate with Python on the PC via the USB Serial connection by writing a pre-determined String Format:
"Data, xTilt, yTilt, zTilt , swState\n"
which is a String starting with "Data", the tilt values separated by commas, the switch State and a newline character.

So if the xTilt, yTilt, zTilt and swState where 2, 5, -1 and 0 respectively the string sent would be:
"Data,2,5,-1,0\n"

As all the data goes out over the Serial Port you can use the Serial Monitor <Ctrl+Shift+M> to view it.

5.
Go to the Next Step, copy the code into a blank Sketch and Upload to the Arduino.

Step 5: Full Arduino Code

/* Sense the Tilting action of an ADXL335 Accelerometer and
outputs the data on the Serial port (USB).
(c) Anthony Kelly, 2013 */

/*
A0 : X-Axis
A1 : Y-Axis
A2 : Z-Axis
*/
int xpin = A0, ypin = A1, zpin = A2;
int swpin = 0;
int swState = 0;
int xAve, yAve, zAve;
int xTilt, yTilt, zTilt;

void setup() {
  Serial.begin(9600);        
  // Fast Baud rate to reduce lag
  pinMode(swpin, INPUT);
  digitalWrite(swpin, HIGH);  
  // Enable Pullup on Switch Pin
// Calibrate the sensor for LEVEL position by taking the Average of 8 readings
xAve = average(xpin);
yAve = average(ypin);
zAve = average(zpin);
}

void loop() {
  xTilt = analogRead(xpin)-xAve;
  yTilt = analogRead(ypin)-yAve;
  zTilt = analogRead(zpin)-zAve;
  swState = digitalRead(swpin);      
  // swState = 0 when Pressed
// Send the Data as a Serial String as follows:
// "xTilt, yTilt, zTilt, swState \n"
// This String will be read by Python with lines separated by '\n'
  Serial.print(xTilt, DEC);
  Serial.print(",");
  Serial.print(yTilt, DEC);
  Serial.print(",");
  Serial.print(zTilt, DEC);
  Serial.print(",");
  Serial.print(swState, DEC);
  Serial.println();
}


// Get the Average of 8 readings from 'pin'
int average(int pin) {
  int Ave = 0;
  for (int i=0; i<8; i++) {
    Ave = Ave + analogRead(pin);
  }
  return Ave/8;
}

Step 6: Python Mouse Control

We'll be using Python3.2 with the pywin32 library to gain control over some of the Windows functions such as Mouse Control. The pyserial library will allow us to read what Arduino is sending us on the USB Serial Port.

The full code is shown in the Next Step. For now let's see the main ideas that make the code work.

1.
At the start we open the serial port:
ser = serial.Serial(COM-1, 9600)   # Open the Serial Port

But first we need to figure out which COM port it's on.
To do that we try each one in turn. If it fails we deal with the exception and keep looking.

# Find out which COM port the Arduino is on
for com in range(15):
    try:
        ser = serial.Serial(com, 9600)    
# Open the Serial Port
       
print("Found on COM",com+1)    # If you got here the COM port was opened OK
        break                                                  # We found it so stop searching
    except :
         print("Nothing on COM",com+1)



2.
swState is the variable we will read the switch state into (from the Serial port from Arduino).

Actually we don't want to know the switch state. We want to know when the button has been pressed.
We can do this by keeping track of the Previous value and subtracting the present value from it. We put that in a function:

def swPressed():
    return swStateD1 - swState
  # 0,1,-1 : Same, Just Pressed, Just Released


i.e.
swState :          1,1,1,0,0,0,1,1,-
swStateD1 :     -,1,1,1,0,0,0,1,1
swPressed():   -,0,0,1,0,0,-1,0,

As the comment says : when the button is pressed swPressed()  returns 1


3.
We read the Serial data being sent by the Arduino using ser.readline() as follows:
This line reads a line up to and including the 'newline' characters '\r\n'.
    rx = ser.readline().decode().split('\r\n')

and we split it based on commas
    data = rx[0].split(',')
Finally we index each piece of data fom data (a List):
xTilt = int(data[0])
yTilt = int(data[1])
zTilt = int(data[2])
swState = int(data[3])


4.
Next we calculate the x,y position for the cursor based on the "centre" position (i.e the cx,cy position of the cursor when we pressed the switch to Activate cursor control - more on that later), and the tilt. We also give some control over the sensitivity i.e. how much the x,y values move for a given Tilt value:
x = cx + xTilt/Sensitivity

And we limit the x,y values. Nicely done in Python as follows:
x = max(min(xMax, x), 0)    # Limit to the Screen co-ordinates

5.
Finally we check if the switch went from NOT Pressed to Pressed:
sw = swPressed()            # Check the Button

If it was just pressed and we are in control of the cursor (enCursor ==1) then we disable the cursor and Break out of the while loop ending the program. But before we Break we emulate the pressing of the SPace Key to PAUSE Flight Simulator as follows:
win32api.keybd_event(0x20, 0,0,0)    # PRESS the SPACE Key to Pause Flight Simulator
win32api.keybd_event(0x20,0 ,win32con.KEYEVENTF_KEYUP ,0)

Otherwise if the switch was just pressed and we are NOT in control of the cursor (enCursor ==0) then we have to enable the cursor (enCursor==1), update the cx, cy variables to the present position
(cx,cy) = win32api.GetCursorPos()
we get the cx, cy values  so we can move the cursor + or - of this point in x and y as the accelerometer is tilted.

and CLICK the LEFT Mouse Button using the win32 library:
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,int(x),int(y),0,0)
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,int(x),int(y),0,0)


Now if we're in control of the cursor, we move the cursor to the x,y position we calculated earlier from the Tilt:
if enCursor: win32api.SetCursorPos((int(x),int(y)))


=================

Copy the code in the next step into a New Window in IDLE (Python IDE) and save it as mouse.py

Step 7: Python Code

import win32api, win32con
import serial


# Initial Cursor Position
cx = 0
xMax = 1900
cy = 0
yMax = 1000

Sensitivity = 0.1       
# Higher numbers make the mouse movement less sensitive to Tilt

# Find out which COM port the Arduino is on

for com in range(15):
    try:
        ser = serial.Serial(com, 9600)    
# Open the Serial Port
        print("Found on COM",com+1)    # If you got here the COM port was opened OK
        break                                                  # We found it so stop searching
    except :
         print("Nothing on COM",com+1)


ser.flushInput()                # Discard Buffer Contents

enCursor = 0      # Start with the Cursor Disbled
swState = 0       # The State of the Switch
swStateD1 = 0     # The State of the Switch the Last Time

def swPressed():
    return swStateD1 - swState 
# 0,1,-1 : Same, Just Pressed, Just Released
   

while 1:

    # Read a line and separate the newline characters from the rest
    rx = ser.readline().decode().split('\r\n')
#    print(rx)   # Debug
    # Now separate the data from the commas
    data = rx[0].split(',')
   
    xTilt = int(data[0])
    yTilt = int(data[1])
    zTilt = int(data[2])
    swStateD1 = swState
     # Remember the old swState before we update it
    swState = int(data[3])

    x = cx + xTilt/Sensitivity        # x-Cursor = centre value (when button was pressed) + Tilt value
    x = max(min(xMax, x), 0)       # Limit to the Screen co-ordinates
    y = cy - yTilt/Sensitivity      
# y-Cursor = centre value (when button was pressed) + Tilt value
    y = max(min(yMax, y), 0)      # Limit to the Screen co-ordinates

    sw = swPressed()            # Check the Button
   
# Disable the cursor if it's Enabled now and the switch is pressed
    if (enCursor and sw==1):
        enCursor = 0

        win32api.keybd_event(0x20, 0,0,0)    # PRESS the SPACE Key to Pause Flight Simulator
        win32api.keybd_event(0x20,0 ,win32con.KEYEVENTF_KEYUP ,0)

        break
# Enable the cursor if it's Disabled now and the switch is pressed
    elif (~enCursor and sw==1):
        enCursor = 1

# Click the LEFT-Mouse Button

        (cx,cy) = win32api.GetCursorPos()
        win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,int(x),int(y),0,0)
        win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,int(x),int(y),0,0)

  
    if enCursor: win32api.SetCursorPos((int(x),int(y)))


# We're Done. Close the Serial Port
ser.close()

Step 8: Ready for Takeoff !!

Here's how it all works.

1. Plug in your Arduino and upload your Arduino code (if you haven't done that already).

2. Start Python IDLE and open mouse.py in theIDLE editor

3. Check that the COM variable in the Python code is set to the COM port used by the Arduino IDE

4. Press F5 to Start the Python code running

5. Open Google Earth

6. Start the Flight Simulator mode (Tools-> Enter Flight Simulator) or (Ctrl+Alt+A)

7. Best to Select the SR22 Aircraft to start with. It's easier to control.

8. Centre your mouse in the Google Earth Screen

9. Press the Button on  the Arduino to Take Control of the Mouse

10. Tilt your Arduino to Control the Plane. Backwards to Climb, Forwards to Descend etc...
KeyBoard Controls:
PgUp : Speed ++
PgDn : Speed--
Space: Pause
More details in Google Earth Help

11. Press the Button on the Arduino again to Relinquish control of the Mouse, Pause Flight Simulator Mode and Terminate the Python Program.

Have fun!!

Here's a Video of me flying...




Step 9: Next Steps

Why not use the ideas from this Instructable to do something else?

* Use a "pattern" of motion on the Accelerometer to start a Program on your PC
- Different "swipe patterns" could work like Hot Keys to perform a function e.g. CUT, PASTE etc..

* Use Python and matplotlib to graph the movement of the Accelerometer and display it on the screen

* Use "SensorMonkey" to plot the Accelerometer movement on the Internet
https://sensormonkey.eeng.nuim.ie/

Comments

akellyirl (author)2013-02-23

I've had some questions on how to add more buttons.

Here's what you need to do:

Add them on the board and wire them up, as with the one in the project. Connect them to some digital pins e.g. 7,8,9.

See where swpin and swState are used in the Arduino code?

Duplicate those lines and edit to represent a new pin/state e.g.
int  swpin1 = 7;
swState1 = digitalRead(swpin1);  
....

pinMode(swpin1, INPUT);
digitalWrite(swpin1, HIGH);

...etc...

Send that as a Serial.print(swState1, DEC); to Python.

In Python; again make a new switch state e.g.: swState1 = int(data[4]) etc....
And duplicate and edit this section so it executes if swState1 is TRUE;
win32api.keybd_event(0x20, 0,0,0)
win32api.keybd_event(0x20,0 ,win32con.KEYEVENTF_KEYUP ,0)


That code section above specifies which key should be pressed and then released (0x20 = 32 decimal = SPACE in ASCII). If you want to activate other keys stick there ASCII Hex code in there instead.

Good luck with it.

MangeshD (author)2016-05-14

Hi

I hv installed pyton32 bit on 64bit system

also installed pywin and pyserial

but after runing mouse.py file which i copied from this page

https://www.instructables.com/id/Flight-Simulator-w...

I am getting this error

Nothing on COM 1
Nothing on COM 2
Nothing on COM 3
Nothing on COM 4
Nothing on COM 5
Nothing on COM 6
Nothing on COM 7
Nothing on COM 8
Nothing on COM 9
Nothing on COM 10
Nothing on COM 11
Nothing on COM 12
Nothing on COM 13
Nothing on COM 14
Nothing on COM 15
Traceback (most recent call last):
File "C:\Users\MANGESH\Desktop\m.py", line 33, in <module>
rx = ser.readline().decode().split('\r\n')
NameError: name 'ser' is not defined

akellyirl (author)MangeshD2016-05-19

The Python program has not been able to read from a device on any of the 15 COM ports it tried. Then it gives up with the "NameError".

Check the COM port shows up in the Windows Device Manager.

Make sure the Arduino code is loaded and running.

Try checking the serial connection from Arduino by using the Serial Monitor in the Arduino IDE. You should see stuff being written from Arduino.

russ_hensel (author)2016-01-23

I love the combination of Python and the Arduino. So I have created a collection about it. I have added your instructable, you can see the collection at: >> https://www.instructables.com/id/Arduino-and-Pytho...

raviteja strange (author)2015-03-01

Traceback (most recent call last):

File "C:\Users\sai kiran\Desktop\mouse.py", line 2, in <module>

import serial

ImportError: No module named serial

Can u please tell me why this error is occuring ?

it will help my main project.....

riddalion (author)2013-10-19

can tweak this to android smartphone?

akellyirl (author)2013-02-24

Added automatic detection of the COM port.

akhandal69 (author)2013-02-23

Amazing,I will try this soon.
btw,is windows necessary ? I have ubuntu 12.10 and windows as well,but can't this be used in ubuntu ?
As Arduino IDE,google earth,python are all available for ubuntu and so it should work.

akellyirl (author)akhandal692013-02-23

Thanks. It should work with ububtu if you have a python api for that. I specified windows because I used the win32 api which is obviously Windows specific.

Good luck with it.

umursengul (author)2013-02-08

I am trying to learn python now (I know, I am a a little late!), and after have a little insight in python in this, I will definitely try this! Thanks :)

akellyirl (author)umursengul2013-02-08

I was a late starter myself with Python. Although very familiar with programming, I was looking for an all-round programming language and I think I found it with Python. I'd recommend trying out the commands you're trying to use in the IDLE command line. Errors show up immediately and you can find the correct constructs for what you're trying to achieve. I always keep : "Programming Python" O'Reilly , near by. Good Luck.

About This Instructable

27,711views

71favorites

License:

Bio: Technologist, Electronic Engineer, sometime Coderdojo mentor.
More by akellyirl:Arduino Guitar Tuner Using DSP TechniquesSmart Home ThermostatArduino Yun Messager
Add instructable to: