Introduction: Using the Complex Arts Sensor Board to Control Pure Data Over WiFi

By plus.google.com/1094500090434189

Have you ever wanted to experiment with gestural control? Make things move with a wave of your hand? Control music with a twist of your wrist? This Instructable will show you how!

The Complex Arts Sensor Board (complexarts.net) is a versatile microcontroller based on the ESP32 WROOM. It has all the features of the ESP32 platform, including built-in WiFi and Bluetooth, and 23 configurable GPIO pins. The Sensor Board also features the BNO_085 IMU – a 9 DOF motion processor that performs onboard sensor fusion and quaternion equations, providing super precise orientation, gravity vector, and linear acceleration data. The Sensor Board can be programmed using Arduino, MicroPython, or the ESP-IDF, but for this lesson we will be programming the board with the Arduino IDE. It is important to note that the ESP32 modules are not natively programmable from the Arduino IDE, but making that possible is very simple; there is a great tutorial here: https://randomnerdtutorials.com/installing-the-esp32-board-in-arduino-ide-windows-instructions/ that should take about 2 minutes to complete. The last piece of setup we need is the driver for the USB-to-UART chip on the Sensor Board, which can be found here: https://www.silabs.com/products/development-tools/software/usb-to-uart-bridge-vcp-drivers . Just pick your OS and install, which should take about 2 more minutes. Once that's done, we're good to go!

[This lesson does not assume any familiarity with either Arduino or Pure Data, however it will not cover their installation. Arduino can be found at aduino.cc. Pure Data can be found at puredata.info. Both sites have easy-to-follow instructions for installation and setup. ]

Also... the concepts covered in this tutorial, such as setting up UDP connections, programming the ESP32 with Arduino, and basic Pure Data patch building - are building blocks that can be applied for countless projects, so don't stoop here once you've got these concepts down!

Supplies

1. Complex Arts Sensor Board

2. Arduino IDE

3. Pure Data

Step 1: Examining the Code:

First, we’ll look at the Arduino code. (The source is available at https://github.com/ComplexArts/SensorBoardArduino .It is recommended that you follow along with the code as we go.)We need some libraries, one of which is not a core Arduino library, so you may need to install it. This project relies on the SparkFun_BNO080_Arduino_Library.h file, so if you don’t have that, you’ll need to go to Sketch -> Include Library -> Manage Libraries. Type in “bno080” and the afore-mentioned library will appear. Press install.

The other three libraries that are used should come with Arduino by default. First, we will use the SPI library to communicate with the BNO. It is also possible to use UART between the ESP32 and the BNO, but since SparkFun already has a library that uses SPI, we’ll stick with that. (Thanks, SparkFun!) Including the SPI.h file will allow us to select which pins and ports we want to use for the SPI communication.

The WiFi library contains the functions that allow us to get on to a wireless network. WiFiUDP contains the functions that allow us to send and receive data over that network. The next two lines get us on the network – enter your network name and password. The two lines after that specify the network address and port to which we’re sending our data. In this case, we will just broadcast, which means send it out to anybody on our network who’s listening. The port number determines who’s listening, as we’ll see in a bit.

These next two lines create members for their respective classes so we can easily access their functions later.

Next, we assign the proper pins of the ESP to their respective pins on the BNO.

Now we set up the SPI class member, also setting the SPI port speed.

Finally we get to the setup function. Here, we’ll start a serial port so we can monitor our output that way if we want to. Then we begin WiFi. Notice that the program waits for a WiFi connection before proceeding. Once WiFi is connected, we begin the UDP connection, then print our network name and our IP address to the serial monitor. After that we start the SPI port and check for communication between the ESP and the BNO. Lastly, we call the function “enableRotationVector(50);” as we will only be using rotation vector for this lesson.

Step 2: The Rest of the Code....

Before we go to the main loop(), we have a function called “mapFloat.”

This is a custom function that we have added in order to map, or scale, values to other values. The built-in map function in Arduino only allows integer mapping, but all our initial values from the BNO will be between -1 and 1, so we will have to manually scale them to the values we really want. No worries, though – here’s the simple function to do just that:

Now we come to the main loop(). The first thing you’ll notice is another blocking function, like the one that makes the program wait for a network connection. This one halts until there is data from the BNO. When we start receiving that data, we assign the incoming quaternion values to floating point variables and print that data to the serial monitor.

Now we need to map those values.

[ A word about UDP communication: data is transferred over UDP in 8-bit packets, or values from 0-255. Anything over 255 will be pushed to the next packet, adding to its value. Therefore, we need to make sure there are no values over 255.]

As mentioned before, we have incoming values in the range of -1 – 1. This doesn’t give us a lot to work with, since anything below 0 will be cut off (or show up as 0) and we can’t do a ton with values between 0 – 1. We first have to declare a new variable to hold our mapped value, then we take that initial variable and map it from -1 – 1 to 0 – 255, assigning the result to our new variable called Nx.

Now that we have our mapped data, we can put our packet together. In order to do that, we must declare a buffer for the packet data, giving it a size of [50] to make sure all the data will fit. We then begin the packet with the address and port we specified above, write our buffer and 3 values to the to packet, then end the packet.

Lastly, we print our mapped coordinates to the serial monitor. Now the Arduino code is done! Flash the code to the Sensor Board and check the serial monitor to make sure everything is working as expected. You should see the quaternion values as well as the mapped values.

Step 3: Connecting With Pure Data...

Now for Pure Data! Open Pure Data and start a new patch (ctrl n). The patch that we’ll create is very simple, having only seven objects. The first we’re going to create is the [netreceive] object. This is the bread and butter of our patch, handling all the UDP communication. Notice there are three arguments for the [netreceive] object; the -u specifies UDP, the -b specifies binary, and 7401 is of course the port that we are listening on. You can also send the “listen 7401” message to [netreceive] to specify your port.

Once we have data coming in, we need to unpack it. If we connect a [print] object to [netrecieve], we can see the data initially comes to us as a stream of numbers, but we want to parse those numbers and use each one for something different. For example, you may want to use X-axis rotation to control the pitch of an oscillator, and the Y-axis for volume, or any number of other possibilities. In order to do that, the data stream goes through an [unpack] object that has three floats ( f f f ) is its arguments.

Now that you’re this far, the world is your oyster! You have a wireless controller that you can use to manipulate anything you want in the Pure Data universe. But stop there! Besides Rotation Vector, try the accelerometer or the magnetometer. Try using special functions of the BNO such as “double tap” or “shake”. All it takes is a little digging in the user manuals (or the next Instructable…).

Step 4:

What we've done above is set up communication between the Sensor Board and Pure Data. If you want start having more fun, hook up your data outputs to some oscillators! Play with volume control! Maybe control some delay times or reverb! the world is your oyster!