Using Machine Learning and YOUR FACE to Control Scratch

by jaymesdec

5.9K610

Intro: Using Machine Learning and YOUR FACE to Control Scratch

In this Instructable, I am going to show you how to control a character in Scratch using your face (with the help of machine learning algorithms)!

This project relies on a few pieces of open source software, including Wekinator, Scratch_OSC_Bridge, and a version of Face Tracker OSC. So we are standing on the shoulders of Rebecca Fiebrink, Kasper Kamperman, and Gene Kogan and many others.

Thank you!

STEP 1: Download and Install All the Required Software

Wekinator (I used a special version from this Machine Learning for Artists and Musicians Class. But the regular Wekinator will work fine)

Scratch_OSC_Bridge

Face_OSC (Mac Install only for now!)

This will only work with Scratch 1.4

Hopefully, someone will make a Scratch 2.0 OSC Extension. If you know of one, please let me know in the comments!

STEP 2: Decide What Data You Want to Collect About Your Face

The Face_OSC software will send a stream of 14 data points (inputs) about your face to Wekinator. Here is what you can get (in order of how they stream):

  1. X position
  2. Y position
  3. Scale
  4. X orientation
  5. Y orientation
  6. Z orientation
  7. Mouth width
  8. Mouth height
  9. Left eyebrow height
  10. Right eyebrow height
  11. Left eye openness
  12. Right eye openness
  13. Jaw openness
  14. Nostril Flare

I tried to take all of this data and send it to Scratch, but the OSC Scratch Bridge crashed, so I recommend starting with just a few of the data points. I got this working well with these 8 data points:

X position (Input 1)
Y position Scale (Input 2)
Left eyebrow height (Input 9)
Right eyebrow height (Input 10)
Left eye openness (Input 11)
Right eye openness (Input 12)
Jaw openness (Input 13)
Nostril Flare (Input 14)

STEP 3: Make a Chart to Plan Out Your Data Collection

I made a simple chart to track what training data I wanted to collect and the minimum and maximum values that I wanted to use for my output range. For example, I know that the x position of my face will be Input 1 from Face_OSC and I know that the minimum and maximum X positions in Scratch are -240 and 240, so I mapped that input to that range. My chart looked like this:

STEP 4: A Bit About Wekinator

I am not going to get too deep into how to use Wekinator. This is just one simple application. But it is really powerful and a great tool for tinkering and learning about Machine Learning.

From the website:

"Wekinator is free, open source software originally created in 2009 by Rebecca Fiebrink. It allows anyone to use machine learning to build new musical instruments, gestural game controllers, computer vision or computer listening systems, and more.

The Wekinator allows users to build new interactive systems by demonstrating human actions and computer responses, instead of writing programming code."

If you want to learn more about using Wekinator (and you will!), I highly recommend this class called, Machine Learning for Musicians and Artists. This Instructable was inspired by my second assignment for the class.

STEP 5: Configure Wekinator for Collecting Data and Training Your Models

Open Wekinator.

Use the exact settings that are in the image above.

The only exception is if you want to use more or less than 8 data points from the OSC_Face tracker. If you do, you will need to change the # of outputs.

When you are done, click Next.

Start Wekinator listening on Port 6448

STEP 6: Setup Your Wekinator for Collecting Data and Training Your Model

Next you will name all of your outputs to match the data that you are collecting for each model and set the minimum and maximum values from your chart.

Click the Edit button for output-1

Click the Edit button next to the Name: outputs-1

Rename the output with the name of the data that your are collecting. I know that Input 1 from OSC_Face will be the X position, so I renamed it XPos.

Set the range of values (minimum and maximum) that you want the input to map to for your outputs.

I used Integers for inputs that mapped to integer values (x, y positions in Scratch)

I used Real-values (floats) for outputs that mapped to float values (scale or size)

Click OK

Next you have to specify which Input will connect to each output. For XPos, I unchecked all the boxes in the list except input-1.

Click Apply Changes after completing setup for each of your Input / Output mappings.

STEP 7: Train Your Models!

Startup OSC_Face. You should see your face bring mapped in real time from your webcam.

In Wekinator, you should see a green indicator for OSC In on the top left of the window. That means that Wekinator is getting data from OSC_Face.

Setup your windows so that you can see both the Wekinator outputs and the Face_OSC display.

Let's start by training the models for mapping Input 1 and 2 to the X and Y Positions.

  1. Turn off "Enable recording of new training data" and "Enable running on new data" for all of the Models except the X pos and Y Pos. These are the red circle and green triangle buttons for each Model.
  2. Slide the X pos and Y pos Model sliders all the way to the maximum.
  3. Move your face to where you want to be when the Scratch character is in the top right corner of the stage. This is a bit confusing because my webcam is reversed.
  4. Click Start Recording. Wait a few seconds while Wekinator collects data about the position of your face.
  5. Stop Recording
  6. Move the X and Y Pos sliders all the way the minimums.
  7. Move your face to where you want to be when the Scratch character is in the bottom left corner of the stage
  8. Click Start Recording.
  9. Wait a few seconds while Wekinator collects data about the position of your face.
  10. Stop Recording
  11. Repeat this for the other two corners of the screen. Remember to adjust your X and Y Pos sliders to the correct positions before you start recording data.
  12. Repeat this same procedure with each of your models.
  13. Record data for the minimum position and the maximum positions for each input

Some Tips on Training Models

  • Remember to turn off "Enable recording of new training data" and "Enable running on new data" for all of the Models except the model or models that you are training.
  • If you mess up, you can always delete the last recording or scrap all your data and start again.
  • I collected about 50-100 data points for each position
  • I found it easier to train some models in pairs (x and y positions, eyes, eyebrow), and others easier to train by themselves (jaw openness, nostril flare).

When you are done collecting data for each model, click Train and wait for Wekinator to do the Neural Network sorcery.

STEP 8: Test Your Models

Now click Run to run your models.

You should be able to see your Model sliders move around and correspond to your face movements.

I found it easier to test models in pairs or individually. You can turn off "Enable running on new data" for all of the Models except the ones that you are testing.

If you are not happy with the responsiveness of any of your Models, you can Record more data for those models to improve them.

Remember to retrain your models after collecting more data!

When you are happy with all of your models, save your Wekinator project and quit Wekinator.

STEP 9: Setup Scratch_OSC_Bridge and Enable Remote Sensor Connections

Start Scratch_OSC_Bridge

Start Scratch 1.4

In Scratch 1.4, go to the Sensing Blocks and Right Click (Control+click) on the Slider Sensor Value block.

Choose "enable remote sensor connections"

Click OK

Check the top console in Scratch_OSC_Bridge to confirm that it is "Connected with Scratch"

In the top half of the Scratch_OSC_Bridge, above ADD PORT, type in 12000. Click ADD PORT. This gets the Scratch_OSC_Bridge to listen to Wekinator on port 12000. The top console should indicate that it is Listening for OSC data at port: 12000.

STEP 10: Start Your Wekinator Project and Face_OSC

Start Wekinator and open your Wekinator project that you trained earlier.

Start listening on port: 6448

Start Face_OSC

In Wekinator, you the "OSC In" indicator in the top left should be green. That means you are getting data from Face_OSC.

STEP 11: In Scratch_OSC_Bridge Choose /wek/outputs As Received OSC Address

In the Scratch_OSC_Bridge, under RECEIVED OSC ADDRESSES:

Click /WEK/OUTPUTS

You should now see /WEK/OUTPUTS under SEND THESE OSC ADDRESSES TO SCRATCH:

This tells the Scratch_OSC_Bridge to send any data from Wekinator with that address to Scratch.

STEP 12: Retrain and Then Run Your Wekinator Models

In Wekinator, click Train to retrain all your models with your data. You don't have to collect more data, you are just retraining and setting up your models again.

Click Run

STEP 13: Check Scratch_OSC_Bridge to Confirm That Data Is Coming Through

Go back to the Scratch_OSC_Bridge to confirm that data is coming through on the console. If not, make sure you followed all the steps in order.

I found that it only worked if I start with all the programs closed. Then followed the previous steps exactly. Here they are summarized.

  1. Open OSC Scratch Bridge
  2. Open Scratch 1.4
  3. Enable Remote Sensor Connections
  4. In OSC Bridge, add 12000 to listen port
  5. Open Wekinator Project
  6. Start FaceOSCWekinator
  7. Start listening to port 6448 in Wekinator
  8. In OSC Bridge Choose /wek/outputs as Received OSC Address
  9. Retrain the models in Wekinator
  10. Run the models
  11. Check OSC Bridge to see if data is coming through

STEP 14: Check to See If the Data Is Available in Scratch

In your Scratch 1.4 application, you should be able to see the output data options in the Sensor Value block dropdown list.

If you don't see the values, try quitting all the applications and starting again from Step 9 in this Instructable.

STEP 15: Create Variables in Scratch for Each of Your Data Points

This is optional but very helpful.

Create a new variable for each data point and set them equal to the corresponding data from Wekinator. See image for an example.

You will have to put that whole block into a Forever loop so that the values continually update.

STEP 16: Draw a Head for Your Character and Make It Follow Your Xpos and Ypos

STEP 17: Make a Nose Sprite

Create and draw a nose sprite. Give it a script that makes it follow xpos and ypos.

You will have to offset the X and Y positions of the nose to get it into the right place. I did this with all the sprites through trial and error, testing as I went along.

STEP 18: Make Your Eye Sprites

Do the same with each eye. Add a set size for scaling to make the eyes smaller and bigger. I found that my eyes were getting a bit big, so I scaled down the size by 80%.

I created the left eye and scripts and then duplicated that sprite, reflected the costume, and adjusted the offsets for the right eye.

STEP 19: Repeat This Process for the Eyebrow Sprites

I found that my eyebrows were going too hight so I scaled down the value by .5.

STEP 20: Finish With the Mouth Sprite

Save your Scratch project!!!

STEP 21: Have Fun Playing Around!

I just learned about all of this in the last 24 hours from the fantastic online course, Machine Learning for Musicians and Artists. This Instructable is not meant to teach any of the details about how this works. For instance, I did not even tell you what OSC stands for! (look it up)

I was just so excited about this that I wanted to get an Instructable out into the world ASAP.

It is clear this is just a start. I did not get the nostril flare working yet (I think I need to add more nostril data and retrain that model). But I hope that you can appreciate the possibilities!

Please offer feedback in the comments and I will try to update, clarify, and improve the Instructable as needed.

I'll share my Scratch and Wekinator projects in the Files for this Instructable, but I'm not 100% sure if they will work for you. But they might...If you do all the steps in the correct order.

I trained my models in good lighting conditions and found that trying to run the program in different lighting conditions did not work that well. So beware of changing ambient light!

To recap. Start with all the programs closed.

  1. Open OSC Scratch Bridge
  2. Open Scratch 1.4
  3. Open the Scratch Project, ExpressiveFaceControl (that should automatically enable remote sensor connections)
  4. In Scratch OSC Bridge, add 12000 to listen port
  5. Open the Wekinator Project, ExpressiveFaceControlContinuous
  6. Start listening to port 6448
  7. Start Face_OSC
  8. In Scratch OSC Bridge Choose /wek/outputs as Received OSC Address
  9. Retrain the models in Wekinator
  10. Run the models
  11. Check OSC Bridge to see if data is coming through
  12. Click the green flag to start your scratch program!