Real Working Harry Potter Wand Using Computer Vision

"Any Sufficiently Advanced Technology is Indistinguishable from Magic" - Arthur C. Clarke

A few months back my brother visited Japan and had real wizarding experience in the Wizarding World of Harry Potter at the Universal Studios made possible through the technology of Computer Vision.

At the Wizarding World of Harry Potter in Universal Studios the tourists can perform "real magic" at certain locations(where the motion capture system is installed) using specially made wands with retro-reflective beads at the tip. The wands can be bought from a real Ollivander's Shop which are exactly like the one's shown in the Harry Potter Movies but do remember: "It is the wand that chooses the wizard" :P

At those certain locations if the person performs a particular gesture with wand, the motion capture system recognizes the gesture and all gestures correspond to a certain spell which causes certain activities in the surrounding area like turning on the fountain etc.

So, in this Instructable I will show how you can create a cheap and effective motion capture system at home to perform "real magic" by opening a box with the flick of your wand :D using just a normal Night Vision Camera, some electronics, and some python code using the OpenCV Computer Vision library and Machine Learning!!!

Step 1: The Basic Idea and the Parts Required

The wands which are bought from the Wizarding World of Harry Potter in Universal Studios, have a retroreflective bead at their tip. Those retroreflective beads reflect a great amount of infrared light which is given out by the camera in the motion capture system.So, what we humans perceive as a not-so-distinctive tip of the wand moving in the air, the motion capture system perceives as a bright blob which can be easily isolated in the video stream and tracked to recognize the pattern drawn by the person and execute the required action. All this processing takes place in real time and makes use of Computer Vision and Machine Learning.

A simple Night Vision camera can be used as our camera for motion capture as they also blast out infrared light which is not visible to humans but can be clearly seen with a camera that has no Infrared Filter. So, the video stream from the camera is fed into a raspberry pi which has a python program running OpenCV which is used for detecting, isolating and tracking the wand tip. Then we use SVM(Simple Vector Machine) algorithm of Machine Learning to recognize the pattern drawn and accordingly control the GPIOs of the raspberry pi to perform some activities.

The Required Materials:

1) A Raspberry Pi 3 model B and the required accessories like keyboard and mouse

2) Raspberry Pi NoIR(No Infrared) Camera Module

3) Harry Potter Wand with retroreflector at the tip: Don't worry if you don't have one. Anything with a retroreflector can be used. So, you can use any wand-like stick and apply retroreflector tape, paint or beads at the tip and it should work as shown in the video by William Osman: Watch The Video

4) 10 Infrared LEDs

5) A 3D printer and PLA filament of your choice

6) 12V - 1A Wall Adapter and DC jack

7) A servo motor

8) An Old Box and a Spoke from the wheel of a cycle

9) A hot glue gun

10) Printouts of some Harry Potter related logos and images on Glossy Sheet of paper

11) Green and yellow velvet sheets.

NOTE: I also tried using an old regular webcam for night vision by removing its Infrared Filter but ended up damaging/displacing its lens which affected the video quality greatly and I could not use it. But if you want to give it a shot, you can go through this great instructable Click Here

Step 2: Install the OpenCV Module!!

Now its time for the first and probably the longest step of this whole project: The installing and building of the OpenCV module in your Raspberry Pi.

The installation of the dependencies for the OpenCV module does not take much time but the build process can take up to 2 to 3 hrs!! So, Buckle Up!! :P

There are many tutorials online which you can follow to install the OpenCV 4.1.0 module. Here is the link of the one I followed: Click Here

NOTE: I strongly recommend installing the OpenCV module in virtual environment as shown in the tutorial as it will prevent various kinds conflicts that can happen due to installing dependencies of different modules or while working with different versions of python.

Step 3: Printing the Camera Mount

The NoIR picamera has no infrared filter, hence can be used as a night vision camera but it still lacks an infrared light source. All the night vision cameras have their own infrared light source which blast out IR rays in the dark which are invisible to the naked eye but the rays can be seen by the camera with no infrared filter after getting reflected from any object.

So, we basically need an Infrared Light Source and something to mount the camera on. For this I designed a simple 3D model of an object on which we could mount the the camera which will we surrounded by 10 IR leds in a circle. The model was created with SketchUp and printed with Black PLA in about 40 minutes.

Step 4: Finishing the Camera Mount

After printing the model I first sanded it with a 80 grit sand paper and then started placing the IR leds in their holes according to the diagram given above.

I secured the leds in their place with some hot glue and then joined the positive and negative leads of two consecutive leds together and then soldered them to create a series connection of the leds.

The positive lead of one led and the negative lead of the led next to it in the bottom were left un-soldered in order to connect the positive and negative ends from the 12 volt wall adapter.

Step 5: The Trained Machine Learning Model

For the purpose of recognizing the letter drawn by a person, I trained a machine learning model based on the Support Vector Machine (SVM) algorithm using a Dataset of handwritten English alphabets I found here. SVMs are very efficient machine learning algorithms which can give a high accuracy, around 99.2% in this case!! Read more about SVMs

The Dataset is in the form of .csv file which contains 785 columns and more than 300,000 rows where each row represents a 28 x 28 image and each column in that row contains the value of that pixel for that image with an additional column in the beginning which contains the label, a number from 0 to 25, each corresponding to an english letter. Through a simple python code, I sliced the data to get all the images for only the 2 letters (A and C) I wanted and trained a model for them.

I have attached the trained model (alphabet_classifier.pkl) and also the training code feel free to go through it or make any changes for training the model with different letters or try different algorithms. After running the program, it automatically saves the trained model in the same directory where your code is saved.

Step 6: The Code That Makes It All Happen!!

After creating the trained model, the final step is to write a python program for our Raspberry Pi that allows us to do the following:

• Access video form the picamera in realtime
• Detect and track white blobs(in this case tip of the wand which lights up in night vision) in the video
• Start tracing the path of the moving blob in the video after some trigger event(explained below)
• Stop tracing after another trigger event(explained below)
• Return the last frame with the pattern drawn by the user
• Perform pre-processing on the frame like thresholding, noise removal, resizing etc.
• Use the processed last frame for prediction.
• Perform some kind magic by controlling the GPIOs of the Raspberry Pi according to the predcition

For this project I created a Harry Potter themed box which I can open and close using a servo motor which is controlled by the GPIO of the Raspberry Pi. Since letter 'A' stands for 'Alohamora' (one of the most famous spell from the Harry Potter movies which allows a wizard to open any lock!!), if a person draws letter A with the wand, the pi commands the servo to open the Box. If the person draws the letter 'C' which stands for close (as I could not think of any appropriate spell used for closing or locking :P), the pi commands the servo to close the box.

All the work related to image/video processing, like blob detection, tracing the path of the blob, pre-processing of the last frame etc., is done through the OpenCV module.

For the trigger events mentioned above, two circles are created on the real-time video, a green and a red circle. When the blob enters the region within the green circle, the program starts tracing the path taken by the blob after that moment allowing the person to start creating the letter. When the blob reaches the red circle, the video stops and the last frame is passed to a function which performs the pre-processing on the frame to make it ready for predcition.

I have attached the code files in this step. Feel free to go through it and make any changes as you want.

NOTE: I had to create two separate python files working with different python versions, one which imports OpenCV module(Python 2.7) and other which imports the sklearn module(Python 3.5) for prediction after loading the trained model, as my OpenCV was installed for the Python 2.7 version while the sklearn was installed for python 3.5. So, I used the subprocess module to run the file HarryPotterWandsklearn.py(for prediction) from the HarryPotterWandcv.py(for all the opencv work and realtime video recording) and get its output. This way I only have to run the HarryPotterWandcv.py file.

Step 7: The Box Opening Mechanism

I had an old red colored box lying around which I used for this project.

For the Box-Opening mechanism:

1. I hot glued a servo near the rear end of the box on a piece of cardboard near the brim of the box.
2. Then I took a spoke from the wheel of a cycle and hot glued it to the arm of the servo.
3. The other end of the spoke was attached to the lid of the box using a piece of wire.

4. Positive of servo was connected to +5V Pin 2 on Raspberry Pi.

5. Negative of servo was connected to GND Pin 39.
6. Signal of servo was connected to Pin 12

Step 8: Making the Box Harry Potter Themed

For making the Box Harry Potter themed, I printed out some colored images of various things like Harry Potter Logo, The Hogwarts Crest, The Crest of each of the four houses etc. on glossy A4 size sheet and the pasted them on the box at various places.

I also used a Yellow colored velvet sheet to cut strips and pasted them on the lid to give the box the same color them as that of the Gryffindor House. I covered the inside of the lid and the cardboard for servo with green velvet sheet. On the inside of the lid I pasted more symbols and an emblem showing the animals which represent each house of the Hogwarts School.

Then finally I stuffed all my Harry Potter related things in the box which included a Gryffindor muffler, a diary with the Hogwarts uniform and the Elder Wand used in this project :D