Introduction: Throw Fireballs With Your Mind Using OSC and Processing

By www.andrele.com
About: Hi, I'm a designer, hacker, and tinkerer that loves to dive head-first into new things and teach people how not to make the same mistakes that I have. :)

In my last tutorial, I talked about how to make a Bluetooth MindFlex EEG headset and output OSC messages.

When I first started playing with the EEG headset, I wanted to see what I could do with it. So naturally, I experimented with moving my mouse with my mind, and eventually got it to throw fireballs in Street Fighter. To do that, I used Java's Robot class in Processing, which turned out to be very useful for other things as well.

This tutorial is going to show you how to make use of those OSC messages by creating a small Processing application that will interpret the messages, then simulate keyboard commands and mouse movements/clicks to throw fireballs in Street Fighter.

If you are already familiar with Processing and parsing data, or just want to learn how to use Java's robot class from Processing, feel free to skip the first few steps. This tutorial is going to assume that you have very little programming knowledge and will be broken down to the following steps:

  1. Basics of Processing and OSC
  2. Setting up your project
  3. Parsing OSC messages
  4. The Java Robot class
  5. Putting it all together

Disclaimer: This tutorial isn't designed to teach you great programming, but rather to get you to see results quickly and hopefully understand why it works.

If you just want access to the source code, you can find it here: https://github.com/andrele/brainwaveosc-robot

Step 1: What You'll Need

Step 2: Basics of Processing and OSC

What is Processing?

Processing is a free open-source programming language that is designed to be easy to use and is great for beginners. It is based on Java, so if you are familiar with that, you'll see some of the similarities.

Download it here and install it before we get started.

How Processing works
Programming is made up of several concepts. I want to introduce you to the following:

  • Libraries - These are chunks of code pre-made by another programmer
  • Variables - These are containers that can hold information like numbers
  • Functions - These are chunks of code that you will be creating to do things
  • Objects - These are like variables. They can hold information, but also contain functions called methods which can perform tasks


Cheesy Analogy

Imagine that you are a wizard or witch that can cast spells. These spells are like functions. They are designed to do something like draw a rectangle on screen. You can either write your own spells (functions), or pull them from a library. Libraries are made up of functions written by other spellcasters, or programmers. Variables are like vials of ingredients that you might need to save or manipulate for your spells. Finally, objects are like creatures that you can summon. For example, you might have a Dragon object that you instantiate as a pet called Aeroth. Aeroth has a wingspan of 20ft, which you can access by using "Aeroth.wingspan". Aeroth can also breathe fire if you command him with a method like this: "Aeroth.breatheFire();"


Moving on

Wow, that was a horrible analogy. I hope you're still with me. Anyways, now that you know about functions, Processing is made up of two functions:

  • void setup() - This function runs only once when you first run your application
  • void draw() - This function runs after setup() and repeats over and over again until you stop your application

We'll talk more about this in the next step.

What is OSC?

OSC stands for Open Sound Control and it is an easy way to send messages between applications, both locally, and remotely. It was originally designed to be a high performance communication protocol for music applications because timing is crucial for that kind of thing. Since then, it has been widely used in creative applications to its ease of use.

What is a protocol?

A protocol is a set of rules for communicating information between applications. Think of English as a protocol. If I write in English, and you read in English, then we have established a protocol for communicating.

What is parsing data?

In our situation, parsing data is a fancy word for reading and interpreting the messages received through OSC.

Step 3: Setting Up Your Project

Start by opening up Processing and creating a new sketch. Sketches are what Processing calls projects. This is where you will be programming. You can download the example code here if you don't want to follow along.

Import the OSC Library

Starting from a blank sketch, you'll want to import the OSC library.

You can do that by typing in:

import oscP5.*;

or

Go to Sketch->Import Library->oscP5. If you can't find oscP5, you may need to add it first by going to Sketch->Import Library->Add Library... and search for oscP5.

Once that is done, you'll want to create an empty object for it on the next line by typing:

OscP5 oscp5;

Create your setup and draw functions

This one is easy. Just type the following:

void setup(){

}

void draw(){

}

You've just created the skeleton for your application.

Step 4: Receive OSC Messages

Now that you have your skeleton, let's fill it in and start reading the OSC messages broadcasted by BrainWaveOSC. Make sure BrainWaveOSC is running and your headset is connected. You can tell by the color of the left panel. If it is green, then you're good.


Finding your OSC port

In BrainWaveOSC, look for your OSC port. In the upper left corner, you should see something like: OSC - 127.0.0.1:7771

The 127.0.0.1 is your localhost ip address and 7771 is your port number. You'll need to save this port number for later.

Setting up Processing to receive OSC messages

Now you'll want to add this line inside your setup() function between the curly braces. Refer to the images as a reference. Your setup() function should look like this:

void setup(){

oscp5 = new OscP5(this, 7771);

}

Great! You've just told the application to start listening on port 7771 for incoming OSC messages. Since it's in the setup function, it'll only run once at the beginning.

Receiving the OSC messages

Next we're going to create a function to receive the OSC messages. While setup() only runs once, and draw() runs all the time, the function we'll be creating will only be run when an OSC message is received.

Create a function that looks like this:

void oscEvent(OscMessage theMessage){

// Print the address and typetag of the message to the console
println("OSC Message received! The address pattern is " + theMessage.addrPattern() + ". The typetag is: " + theMessage.typetag());

}

Ok let's break it down. void oscEvent(OscMessage theMessage) says that this function is called oscEvent, and it takes an OscMessage object as a parameter called theMessage. Ignore the void for now, it means the function isn't expected to return a value.

Inside the function, we've got two lines. Lines prefixed with the // just means that it is a comment and will not run. Ignore that first line. The second line is println, which is short for Print Line. This writes messages in the black area below your text editor when you run the application (called the console). The contents of println tell it to print "OSC Message Received!" and uses two methods of OscMessage:

  • Address Pattern - This is like the subject of an email. It's the name of the message
  • Typetag - This tells you what the contents of the message are
    • For example, a typetag of iii means that there are 3 whole numbers inside the message

Run the application

Click on the play button in the upper left corner. You should see a small gray window pop up and the black area below your text editor should fill up with words like:

OSC Message received! The address pattern is /signal. The typetag is: f

If you're here, then you've successfully established a connection with your OSC application!

Step 5: Parse OSC Messages

What we need to do now is extract the numbers from the messages we've been receiving and do something useful with it.

For this, I'm only going to focus on the Attention value coming from BrainWaveOSC. Trent has conveniently named the values with meaningful Address Patterns, so all we need to do is check our incoming messages for it and pull it out.

Inside your oscEvent function, you'll want to add:

if ( theMessage.checkAddrPattern("/attention") == true ) {

println("Your attention is at: " + theMessage.get(0).floatValue());

}

To make sure your console doesn't get flooded with other messages, comment out your println from the last step with // at the beginning so it looks like this:

// println("OSC Message received! The address pattern is " + theMessage.addrPattern() + ". The typetag is: " + theMessage.typetag());

Run your application

Run your application again with the play button in the upper left corner. This time, your console should show something like:

Your attention level is: 0.0

I wasn't wearing my headset when I was testing it so it shows up as 0.0. If you are wearing it, you should be seeing the numbers change.

Add a global variable

Awesome. So now we're receiving just the attention values and printing them to the screen. We need a way to pass that information to another function so that we can do something with it. The easy way to do this is with a global variable. Add this line underneath OscP5 oscp5;

float currentAttention;

Change your oscEvent function to look like this:

if (theMessage.checkAddrPattern("/attention") == true) {

currentAttention = theMessage.get(0).floatValue();

println("Your attention is at: " + currentAttention);

}

Now, you've created a variable that can be accessed from any function and is updated when oscEvent sees an OSC message with the address pattern "/attention".

Step 6: The Java Robot Class

Now you're ready to move things with your mind. We're going to introduce a new class called the Robot class. Sound cool huh? This is something that is built into Java that will allow us to read status information about the keyboard and mouse as well as manipulate them in code.

Import Java libraries

Up at the top of your file, import the Java libraries like so:

import java.awt.*;

This opens up all kinds of cool things to us and allows us to use the Robot class.

Underneath OscP5 oscp5; put this in:

Robot bob;

This creates a placeholder for a Robot, with the name bob.

Breathe some life into the robot

Add this to your setup() function:

try { // Try and create a new robot named bob

bob = new Robot();

}

catch (AWTException e) { // If there is an error, print it out to the console

e.printStackTrace();

}

This one is a bit more complicated to explain. Basically, you're going to try and create a robot object named bob, but if it doesn't work for whatever reason, it will print an error and stop your application - because y'know... robots are dangerous.

Add a threshold

Before we start triggering stuff with our brain, lets set some ground rules. Add this below float currentAttention;

float threshold = 75.0;

This is an arbitrary number that we'll use later to tell our program when to press a key.

Add key presses!

So the final product for this is to throw Hadoukens in Street Fighter with your mind. If you check out the images, you'll notice that the key for that is 'S'. So in your draw() function, put this down:

if (currentAttention > threshold) {

bob.keyPress(java.awt.event.KeyEvent.VK_S);

} else {

bob.keyRelease(java.awt.event.KeyEvent.VK_S);

}

What this does is say that if your current attention level exceeds your threshold, tell bob the robot to press 'S'. Otherwise, release 'S'. This could be mapped to any other key. Letters are just VK_<letter>, but some of the function keys have abbreviations. You can look them up in the Java documentation here.

Step 7: Putting It All Together

The hard part is over! If you've gotten this far, the last step is to try it out.

I used Julian Knebel's Street Fighter code here to test out this app. You can download his repository and try it out, but to make things easy, I hosted it on my server for you. Visit http://andrele.com/street-fighter-css-master/ to try it out!

Adjust the Processing, BrainWaveOSC, and browser windows so that you can see everything. Make sure everything is running properly, run the Processing application, then switch to the browser window.

Your mileage may vary and you may have to adjust the threshold levels to get some decent control out of this. There are probably much better ways of doing this, but the goal of this tutorial was to teach you how to parse OSC messages and use Java's robot class to manipulate your keyboard. The EEG headset was just a fun way of putting everything together.

Learning how to press one button may not sound like much, but there are a lot of one-button games out there. With some creativity, I'm sure you'll be able to figure out plenty of cool things to do with Processing, Robot class and OSC. I hope you enjoyed this tutorial! :)