Air - True Mobile Air Guitar (Prototype)

Okay so,

This will be a really short instructable about the first part of finally getting closer to a childhood dream of mine.

When i was a young boy, I always watched my favorite artists and bands play the guitar immaculately.

As i grew up, I was thankful enough to learn how to play a guitar and even play some owned by others, but still don't have my own :(

So I decided to finally sit down and make one that runs completely on the phone, uses computer vision and lets people like me that want a guitar but might be traveling, broke or too young to get one yet!

You can find the prototype app at this website

To see how to play, go to the "You're Done" Step.

* Be sure to use it on your phone and turn the screen sideways to landscape mode *

Enjoy!

(ﾉ◕ヮ◕)ﾉ*:･ﾟ✧･: *ヽ(◕ヮ◕ヽ)

Supplies

1. Smart Phone

2. Desktop Computer or Laptop (For Programming)

Step 1: Background and Note on Code

This Project is largely a coded project aimed at running entirely on the phone.

In coming up with this project, I tried various other apps and looked up other devices currently on the market like the AirJamz or Kurv guitar, portable guitars or even the Real Guitar app on the play store.

The problems I found lacking in many of them was:

1. Some needed external devices

2. Almost all apps didn't really let you play actual chords or music and were just fret board simulators

3. External Devices were prettypricey and many guitarist recommended just buying an actual guitar

These are pictured in the accompanying images.

And so the Air app has to solve these problems while being entirely able to run on the phone. I believe this is possible because in 2020 we have far better mobile browser technology and a lot of improvements in computer vision that can allow us to do wonders with a single RGB camera.

So I proceeded to do some sketches of what It would look like and how it would function before getting fully started.

I also drew out my coding milestones so in this instructable, instead of boring you with code, I will take you through my design process and attach the annotated code at the end for you to read and look at if you need.

The entire code can be found at https://github.com/msimbao/air and i recommend structuring your code files similar to this.

Also note that for the app to run, it needs to be hosted. I have so far only found it to run when hosted on github. :)

Step 2: Strumming Action

The first major coding milestone, was to find a way to replicate a strum digitally without any external periphery. My immediate thought was to use the RGB front facing camera of my phone.

My thought was that if a person has a chord they want to play, then when they swipe their hand in front of their camera, a sound will be played.

After figuring that out, I then needed a good programming language that could be used to interface well with the RGB camera.

I settled for Javascript because I could make a cross-platform app with React Native or something else or could just host the guitar on a website and it could be available for everyone.

I then found different ways to figure out how to get the hand swipe to trigger an action that could be a chord sound playing but there were many ways to do this.

Machine learning worked super well when i tried IBM's services and trained about 3000 images over a week for swipe recognition as well as chord recognition. I also tried handtrack.js by victordibia. Unfortunately they were both unbelievably slow on mobile phones.

I then stumbled across motion detection and an implementation by the lonekorean at diffcam.com. I learnt that it is possible to use the webcam to record two separate frames and then calculate the difference between the frames and give the difference a score. If that score exceeds a certain threshold, I then execute an action.

The lonekorean also made an engine for his diff cam that I decided to use for the Air guitar and it worked perfectly for getting me the motion score!

Attached are pictures of attempts at training Machine learning models as well as the diffcam.com example I learnt from.

Note: In this current prototype, strumming kinda repeats over and over, to stop it, simply hold the chord you want to play next down. This is a bug that we hope to fix forward.

The code for the full strum is found in the script.js file attached here and the diffcam engine by lonekorean is here.

Step 3: Chord Recognition

The next coding milestone was then to find a way to handle chord recognition live.

I wanted a user to be able to replicate actual chord shapes and so practice good hand placement and also help them practice different chords.

Like in the last step, I tried out Machine learning for chord recognition, but it was very slow on mobile phones.

I then learnt something from the Real Guitar app that it might be possible to place a fretboard on the phone screen use the screen to generate chord shapes.

I then had to learn how to allow multi-touch interaction in javascript and found an awesome tutorial and example from Mozilla's docs

Touch interactions can be tricky especially in Javascript but the idea is that we can create certain divs and then define functions to handle different touch events:

1. touchStart: When a finger touches the screen

2. touchEnd: When the finger leaves

3. touchMove: When the finger is still on the screen but changes position

We then work around those functions to define our own elements that respond to different touch events and combinations.

In our case, we design a fret board using CSS and then using Javascript, tell the app that when certain div's are pressed together, a chord should be recognized.

We can then define an audio object that we will pass the chord to and then play that audio when a swipe event occurs.

To define different chord combinations, I made the fret board using this image and then just set each special position to be a div that I could touch and combine with others.

The code to define chord progression is found here and the fretboard controller is found in the code attached.

Step 4: Finding Chord Sounds

Now that our system is set to recognize, we need some actual chord sounds.

Thankfully, freesound.com always comes to my rescue when I need audio samples. I simply searched for chords and found an amazing pack of major chords by danglada.

I then downloaded them and edited them using audacity to make sure that the sound started immediately rather than the short pause at the start of most of them when they were being recorded.

To clip them using audacity, i simply dragged them into the app then selected the portion of the sound that i want (the entire wavy part and none of the flat line sections that have no sound). I then go to the Edit tab > Remove Special > Trim Audio. Then i got to the Tracks tab > Align Tracks > Start to Zero. I then go to file, then Export > Export as WAV.

I export as WAV because I have found it easier to deal with in Javascript audio projects.

I then used glitch.com to host these files because they have an amazing content delivery network that can be used for different projects that you have. Another option could be to use firebase which is my goto for different projects that might have more information to store like the makerspace inventory app for my college's makerspace.

You simply have to drag and drop the assets to the project directory and then you can find a link when you click the asset folder and click the asset you want to get. Glitch will then produce a unique CDN url for your asset. For example, here's the link to the A major Chord sound.

I can then link all these chords together in a getChord function that will look for when a specific combination of fret positions has been pressed and then assign an appropriate chord for the app to play when a hand swipe event occurs.

Step 5: Finishing and Hosting the Entire App

There are many ways to go about hosting.

Honestly the best I've found is simply to use github. This is because if you have programmed an app well, you can make your entire back end be served by a database or firestore from firebase or even use a CDN from glitch.com and other places to store assets in.

To host the project on github, all you have to do is open a github account, make a new repository. Then to make it easier to set up, after you have put your project name, always make sure to add a license (i'm not an expert but i have found that it makes my life easier). I always just use a public license like the GNU.

Once the repository is set up, we can just drag and drop our files into the repository and click the green commit button at the bottom.

We then go to the Settings tab with the gear icon at the far right of the repository page under the star and watch buttons. Once in the settings, scroll down till you see the Github Pages box. Change Source to master branch and pick a theme if you want. You can learn how to use themes by googling them (I don't ever use them because I often bring my own CSS and theme ideas).

When the page is ready, you will get a green highlight and tick telling you that your site is published and can be accessed.

Step 6: Done

You can now enjoy an awesome jam session in the comfort of your own headphones, bedroom or on the train. Add some more chords if you like and even play around with the guitar fret positions.

A quick Note on Motion Detection

1. The threshold for a guitar strumming swipe can be adjusted in the script.js file but make sure that when you are using the app, the background that your phone sees is relatively still.

2. For example, in the train, it's better to sit down and put your headphones on and turn your phone inward so that if passengers are moving around you the phone camera can only see your hand moving most of the time.

3. The hand clutching the phone needs to be relatively still depending on your threshold. I think I'll run some tests with a high threshold and update the limits going forward to be more specific.

To Play:

Load the app on your web browser, then tilt it to landscape mode.

Then when you swing your hand, a chord will play, however, it will keep playing until you touch the F key in the lowest right corner.

Alternatively, you can stop the sound by making a chord combination.

When you make a chord combination, the current sound stops, then a new chord sound is selected.

Step 7: Things Learnt and Final Words

I really loved working on this project even if it took a long time to prototype and get the app produced while working on other projects and home work. I learnt a couple of awesome stuff along the way too;

1. When designing digital products, always make sure to make your prototypes as fast as possible because your first assumptions will be wrong and you need to run over them quickly to get to the end.

2. Avoid spending money on a project as much as possible. Always reuse whatever you can and always start with simple things you have on hand.

3. Don't be afraid to learn new languages, frameworks and systems. They are often easier than you think at first.

And huge thanks to the lonekorean for making my dreams come true

If ya'll are interested in how the app develops you can join our mailing list. A small team and I will be working to make a full version to help people who are broke, traveling or young kids have access to an awesome portable guitar wherever they are.

We would really love some help especially from graphic designers, guitar players and coders to test and flesh everything out.

Enjoy (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ ･: *ヽ(◕ヮ◕ヽ)