Tech-Magic: Home Automation Using Interactive Wand From Wizarding World of Harry Potter

28,306

61

41

Introduction: Tech-Magic: Home Automation Using Interactive Wand From Wizarding World of Harry Potter

About: Tinkerer, dreamer, nerd. Professionally, I write code for robots (the hardware kind).

Last Christmas I visited the Wizarding World of Harry Potter at Universal, Orlando and was blown away by the detailed replication of their real counterparts in Europe, and also, the magic created by muggles. I purchased one of these 'interactive'/ muggle magic wands that made stuff happen on strategic spots throughout the park, and though it felt like going back to the basics of wandlore, it was a really neat setup! The funny tip of this wand is actually an IR reflector and the magical spots around the park had an IR transceiver system capable of tracking the wand using these reflectors. Now, being the tech nerd that I am, I decided to create some of my own magic using this wand. So follow along if you want to create some too.

Please note that I built this project using stuff I already owned. So some of my choices might not be the best or even feasible for everyone. Therefore, instead of elaborating on my code, I will be explaining each step's logic in detail so you can build your own version using your own tools. But, if you will be using the same tools, then you should be able to do so easily by following the overview.

*Familiarity with coding and setting up IDEs is assumed*

Project Code:https://github.com/sanni-t/techMagicApp

Equipment & tools used:

1. Windows PC (Win 7+)

2. Kinect (v2) for Windows [or XBox One Kinect + windows adapter]

3. Interactive wand from Universal Studios' Wizarding World of Harry Potter

4. Hue Lights(w/ bridge)

5. Spotify premium account

6. A pair of Simblee Boards (this or that)

PLEASE NOTE: This instructable contains moving images (a.k.a gifs in the muggle world) which don't seem to show well on mobile browsers. So it's best viewed on a computer/ tablet

Step 1: Setting Up Visual Studio & Kinect

As done by the engineers at Orlando, we are going to use an IR transceiver to track the wand too. More about how this works in the next step.

So now,

1. Install Visual Studio 2015 (Community version is free)

Project is not tested for other versions so if you plan on using v2017, you might have to modify some of the code.

2. Install Kinect 2.0 SDK.

3. The SDK should come with a program called Kinect Studio. Use it to test your Kinect.

4. Once Kinect is set up, open Visual studio and create new project (C++ console application)

5. Add Kinect dependencies as shown in the video above.

6. Add OpenCV 3.1 (a computer vision library) to the project-

  • Go to: Tools -> NuGet package manager -> Manage NuGet packages for Solution -> opencv3.1 (lookup using search bar)
  • Refer to OpenCV's website to familiarize yourselves with the library.

7. If you are going to connect to http services, you will need to add cURL (a library for transferring data with URLs) the same way-

  • Go to: Tools -> NuGet package manager -> Manage NuGet packages for Solution -> Add:
    • rmt_curl (version 7.51.0)
    • rmt_zlib (version 1.2.8.7)
  • Refer to cURL website for more info on how it works

8. To test your setup, add the header files: KinectHandler.h, ImageProcessor.h and httpService.h from techMagic directory to your project, include them in the main header file and build the project (make sure to not just copy them to your project folder but also add to it by choosing 'Add existing file' in solutions explorer). If you get errors, go back and check if all packages have been installed properly. If everything compiles, congrats! You can proceed to the next steps.

Step 2: Project Outline

There are quite a few IR cameras available in the market, but we need one that has an IR light transmitter too. The Kinect for Windows or Kinect v2 fits this bill perfectly. It has quite a powerful transceiver, giving us a longer range for wand placement, at an affordable price. Note that the older Kinect version won't work because it projects IR light in a dot grid-array (as opposed to Xbox One Kinect's steady IR light, as shown in this blog). This makes it difficult for enough light rays to be incident on the small wand tip and be reflected off of it to be detected by the receiver.

Picture 2 shows how the wand tip looks illuminated in Kinect's IR camera

*Picture 3gives the code outline*

Before proceeding to the next steps copy the file 'config-template.h' to the solutions folder, rename it to config.h and add to your solution. This file will hold all your personal configuration data like ip addresses of wireless devices, api account ids, authentication keys etc. So if you are saving your project to a public repo then you would want to add them to the ignore list.

NOTE: Use the DEBUG macro setting in config.h to change to/from a debug mode (better term- unfinished mode) to select whether you want to use wand movements to trigger actions or use keypresses instead. Useful while setting up device interactions.

Step 3: Image Processing

Picture 1 explains the process of image processing to detect wand movements/ gestures (code source- ImageProcessor.cpp)
Picture 2 shows how blob detection function detects the wand tip as a 'blob'

The problem with using just blob detection is that it picks up reflections from the surroundings as well. So we minimize this interference by performing background elimination (Picture 3). This gets rid of almost all static reflections. Reflections from items like eye glasses and watches would still cause interference though, so, you would want to take them off while testing.

Next, to filter out unwanted blobs that weren't removed in background elimination, we check if the blob is moving at a reasonable speed. Spurious blobs(noise) jump in and out of frame, as opposed to our wand movement, which is expected to follow a pretty constant speed.

Once we eliminate all the noise, we will hopefully get a nice and clean trace of the wand.
NOTE: To make the entire wand detection code work you will need to add ImageProcessor.cpp to your solution, include appropriate header file and class instance in your mainProgram.h (see techMagic.h) and copy spellmodel.yml to the project folder.

Step 4: Spells, Wand Movements and Spell (character) Recognition

Picture 1 shows the wand movements for the four spells we will be using.
This picture serves another important purpose- it's used in opencv's machine learning algorithm to train for character recognition. Upon successful training, the program creates a .yml file which is then referred to while performing character recognition. This blog gives a nice explanation of how this works.

*The ImageProcessor code contains training modules too so you can add your own spells. I will update this page once I clean up this section of code*

Practice the spells until you can draw them convincingly a good number of times so that the algorithm can find a proper match in the trained model.

Update:

I have updated the code to enable 'spell training', that is, train the machine learning algorithm to recognize your spells. To do this, use the brach 'addSpellTraining' from the techMagic repository.

In here you will find some updated options in the config file. The steps for spell training are:

1. Save images to use for spell training..
In the config file, set ENABLE_SAVE_IMAGE to true. Upon running the code now, you will be able to use your wand to draw more freely. It'll feel a lot more like drawing on some high viscosity liquid where the trace will slowly disappear as u keep drawing more. I find that this makes it easier to correct my mistakes by just holding the wand stationary for a few seconds, which erases the trace, and starting the drawing all over again. When you draw a spell satisfactorily, you can hit the SPACE bar to save it to the project directory (you can change this path in the config file). The spell drawing/image will be saved as a 64x64 pixel png image. Save as many images as you would like for each spell. The more you use, the larger the sample set for training would be, which will increase the spell recognition model accuracy. The code uses a default value of 20 images per spell, which, i have found to be a good balance between accuracy and number of samples

2. Create the master image to be used for spell training..
Once you have saved the images for spell training, you would now stitch the images together using either a photo editing software or writing your own script. You can use the above image that I used for spell training as a reference to create your own set. If you don't wan't to replace the spells from my image but just append a few more, just edit the image to add yours to it.

3. Run the training algorithm:
When you have your edited image prepared, switch ENABLE_SPELL_TRAINING in the config file to true and edit the file names for GESTURE_TRINER_IMAGE and TRAINED_SPELL_MODEL if required. Run the code and voila! you should have your .yml model file ready to use in normal mode!

Step 5: Interfacing With Hue Lights

- Go to Hue API documentation page and follow the steps to get your hue's ip address and your username. Add them to config.h as shown in the picture above.
- Add httpService.cpp to your solution. You can also lookup the json data corresponding to different light configurations that you want and make changes in the .cpp file accordingly.
- The default setting is to turn on/off the light numbered 1

Step 6: Interfacing With Spotify

NOTE: Using the play/pause feature described here requires a premium account.
1. Go to Spotify api tutorial and follow all the steps to the end to get your app's authentication string and refresh token. ** IMP: Use the Spotify files provided in the techMagic repo instead of the OAuth example from Spotify repo (Picture 1) **

2. When you get to the last step ('Running the application') on the above tutorial page and log into your account, your auth string and refresh token will be output to the command line interface. Copy these two strings to the corresponding fields in config.h as shown in picture 3 above.

3. To play a specific track/playlist/album, copy its Spotify URI to the SPOTIFY_PLAY_CONTENT field in config.h

4. Test the code by performing the correct spell or by using 'm' keypress in debug mode

Step 7: Blinds Controller

1. Motor used: https://www.pololu.com/product/2365

2. You basically want to attach something to the motor shaft to be able to couple with the blinds's tilter. I just used a spare gear attachment I had and drilled a hole along its axis of rotation.

3. Then I used a bendable metal strip to hold the motor in a slight angle so the tilter is along the motor's rotational axis. It's important to use something sturdy but bendable to allow for the motor to adjust its alignment as required.

4. If you are new to Simblee, I recommend reading its documentation. Basically it is a BLE module which also has its own mesh communication 'SimbleeCOM' protocol. I used this feature to be able to control two devices- the blinds and the toy robot, using one central module.

5. Logic: A central Simblee board, connected to computer, listens to commands sent by techMagicApp and broadcasts them on SimbleeCOM. The blinds' controller Simblee board listens to messages on SimbleeCOM and when it encounters the messages intended for it (e.g., 'TURN_BLINDS'), it moves the motors. When DEBUG is true in techMagic, you can use the keys 1,2,3 to rotate motors 1-3 clockwise, and keys !,@,# to rotate them counterclockwise. This is a handy trick to initialize the blinds' position.

6. The blinds controller schematic is attached below and the code for it and the central Simblee board are in the 'Arduino files' folder in the project repo. The Serial class (Serial.h and .cpp) in techMagic project facilitates serial communication.

*I didn't add the walking toy robot since it's a whole another project and its interface is the same as the blinds controller.

Step 8: Put It All Together

That's all it is! Put it all together and you should have a project folder quite similar to the techMagic repo (sans the documentation pictures). Get your wands out and bring objects surrounding you to life!

End Notes:
Going through the code while following this instructable will help understand the project better. I have tried to make my code quite easy to read with a lot of comments, still, improvements and comments are welcome. This project was only a fun experiment and I thought of sharing it so we can collectively make the world more magical. It's not a turnkey project but I hope creative tech enthusiasts would be able to learn some valuable stuff from it about technology-powered Wandlore ;)

Step 9: Fin

Special thanks to:
..my friends Alexis and Natasha for helping me with documentation
..my roommate Aish, for pointing me towards cURL
..and my cousin, Manasi for the amazing trip to the Wizarding World!

Thank you for reading!!

Wireless Contest

Participated in the
Wireless Contest

1 Person Made This Project!

Recommendations

  • Make it Real Student Design Challenge #3

    Make it Real Student Design Challenge #3
  • Colors of the Rainbow Contest

    Colors of the Rainbow Contest
  • Arduino Contest

    Arduino Contest

41 Comments

0
olliesub103
olliesub103

Question 22 days ago

"Exception thrown at 0x00EF0D04 in techMagic.exe: 0xC0000005: Access violation reading location 0x00000000.
If there is a handler for this exception, the program may be safely continued."

0
sanni-t
sanni-t

Answer 21 days ago

Is there no other explanation or description with that error? The numbers aren't helpful.
Also, were you able to verify that your Kinect works using the camera viewer app like I mentioned before?

0
olliesub103
olliesub103

Reply 21 days ago

There doesn't seem to be any other error messages; and yes, my kinect works with the app.

0
sanni-t
sanni-t

Reply 20 days ago

OK. Good to know the kinect works. But it is strange that there's no explanation for that error.
I got hold of a kinect and tested the code just now and everything works for me. I wonder if it might have something to do with Visual studio 2017 and the way it differs in linking libraries and dependencies from VS 2015. But before declaring this as an environment issue, would you mind sending me your code files? I want to check if they produce the same error on my setup.
(It doesn't look like you can attach files to messages on instructables, so maybe you can upload to some cloud storage and message me the link. Just be sure to remove any credentials you might have added to the config file.)

0
olliesub103
olliesub103

Reply 17 days ago

Will do when I get the chance!

0
olliesub103
olliesub103

4 weeks ago

It says when I run it "Handle was not attatched. COM8 is not avaliable".

0
sanni-t
sanni-t

Reply 4 weeks ago

COM8 is the default serial port used by the simblee board I'm using to control the blinds & the toy robot. If you aren't using those things, you'll have to comment out the lines using the Serial class and botToggle, updateSimblee & blindsToggle functions. Replace the bot & blind functions with print statements or whatever else you want to do when you cast the spells.
(Also, if required, let's continue conversation on this comment thread instead of new ones every time. It's easier to keep track :) )

0
olliesub103
olliesub103

Reply 4 weeks ago

It worked and the error went away. however, when i try to do the 'start without debugging' it opens the wand trace window (which is a grey shade) then closes around .5 of a second after without showing anything. A command prompt says

***** VIDEOINPUT LIBRARY - 0.1995 - TFW07 *****
Press any key to continue . . .

After the window disappears.
I'm really sorry to keep bugging you!

0
sanni-t
sanni-t

Reply 4 weeks ago

What happens when you start in debugging mode?

0
olliesub103
olliesub103

Reply 4 weeks ago

"Exception thrown at 0x00EF0D04 in techMagic.exe: 0xC0000005: Access violation reading location 0x00000000.
If there is a handler for this exception, the program may be safely continued."

0
olliesub103
olliesub103

Reply 4 weeks ago

I got this one afer doing what you said^

0
olliesub103
olliesub103

Reply 4 weeks ago

Hey :D did you see my message? sorry if you are still figuring it out or you are busy with work or university or something lol

0
sanni-t
sanni-t

Reply 4 weeks ago

Do you have the spotify account and hue lights set up? If you aren't using them, you will have to comment out the relevant code from techmagic.cpp & techmagic.h for those as well. Probably replace MusicToggle & lightsToggle with just printlines. Check that you get the correct print output in debug mode. Then disable debug mode and see if the camera view opens up. If everything works in debug mode but not in normal mode, then make sure that your kinect is working correctly. To check this you can use the camera viewer app that comes with the Kinect for windows SDK.
If the Kinect works in the SDK camera viewer but not in the techMagic app and the errors thrown don't have any explanation, then I'm afraid there might be a bug in my application or the kinect driver. As I said, it is quite old at this time and I don't have the time, nor my old Kinect(RIP), to troubleshoot it. I am instead hoping to publish a much easier to use application very soon. So, if this doesn't work, you can try the new one out.
Let me know how your troubleshooting goes. I might not be available to reply until the weekend but will try to respond if it's a quick fix.

0
olliesub103
olliesub103

Reply 4 weeks ago

Welp I commented out the spotify and hue lights lines and opened the debug mode and i got a different error this time:

Exception thrown at 0x009A0D04 in techMagic.exe: 0xC0000005: Access violation reading location 0x00000000.

0
olliesub103
olliesub103

Reply 4 weeks ago

okay! I will try now and update you! thanks :)

0
olliesub103
olliesub103

4 weeks ago

Sorry to bother you again.. I cannot seem to get rid of this error! Do you have any ideas?

Error (LNK2019) unresolved external symbol _GetDefaultKinectSensor@4 referenced in function "public: long __thiscall KinectHandler::init(void)" (?init@KinectHandler@@QAEJXZ) , techMagic , KinectHandler.obj

0
sanni-t
sanni-t

Reply 4 weeks ago

Sounds like the Kinect dependencies might not have been added correctly. You might want to recheck if you have added all dependencies as shown in the instructable. One thing I can think of is verifying the paths of all dependencies according to your platform- x86 vs x64. I believe you'll have to select the appropriate library file paths.

0
olliesub103
olliesub103

Reply 4 weeks ago

Okay! thanks ill check rn!

0
olliesub103
olliesub103

5 weeks ago

I just need Kinect.h but I cant seem to find it....

0
olliesub103
olliesub103

5 weeks ago

I feel so stupid.. I am confused as to how to actually make this? I got the kinect adapter and it took me 3 hours to literally install Visual Studio (which i had to resort to using 2017 version because it just was not working) and I made the project and downloaded the packages and everything and now I am lost.. Please help lol :(