Introduction: Using Kinect Hacks for Dance Training

Picture of Using Kinect Hacks for Dance Training

This is a project we worked on at the Georgia Governor's Honors Program over the summer in our Technology major! We got to pick whatever project we wished so we chose to merge dance and technology using the motion tracking technology of the Kinect and the newly released Microsoft Kinect SDK. The main goal of this project is to alert dancers when they are out of alignment consequently helping them to turn better and faster. We hope you enjoy!

With the exception of the code We instruct you to write, the code is credited to Microsoft and the hacking community!
PROJECT BY: KELAH HENDON AND KELCY NEWTON

Step 1: Downloading Required Software

The main software used/Components required
--Microsoft Visual Studio 2010
--Microsoft Kinect SDK
--Microsoft Kinect Developer Toolkit
--XNA Game Studio/XNA Framework
--XNA Basics (the core of the program)
--Microsoft Kinect

These are all available online. Simply follow the link provided below. Once both the SDK and the tool kit have been downloaded and installed select the tool kit and it should take you to a browser with several hack project options. Here you will need to download the XNA Game Studio as well as the XNA Framework (located at very bottom of hack list). The XNA Basics is in the middle of the list and should be downloaded last!

Opening the Sample in Visual Studio:
For some this step might seem unnecessary, but we have decided to include it regardless. Next for the XNA basics download, click the button that says documentation. This will take you to Microsoft's help website and provides step by step instructions on how to open the XNA Basics file in visual studio,

Link for Downloads: https://www.microsoft.com/en-us/download/details.aspx?id=44561

Note: This project also requires Windows 7 or higher. Microsoft has not released an SDK that works with Vista or XP.

Step 2: Choosing Joints to Analyze

Picture of Choosing Joints to Analyze

The majority of this coding happens in the skeleton stream renderer.
Since we were making this project specifically for dance, We knew we only wanted to look at certain joints.

First, make a list that will contain the joints we do not wish to see where we introduce other variables:
private list NoJoints;

Second, we must initialize the list in the public override void initialize section of coding
NoJoints = new List {JointType.Spine, JointType.ShoulderCenter, JointType.Head, JointType.HandLeft, JointType.HandRight, JointType.HipCenter, JointType.FootRight, JointType.FootLeft, JointType.ElbowLeft, JointType.ElbowRight };

Third, we must tell the program to not draw the joints contained within the list. Go down the code to the place where the joints are drawn. There should be a green comment that says "Now draw the joints":
if (!NoJoints.Contains(j.JointType))
{
this.SharedSpriteBatch.Draw(
this.jointTexture,
this.mapMethod(j.position),
null,
jointColor,
0.0f,
this.jointOrigin,
1.0f,
SpriteEffects.None,
(0.0f) ;
}


The code in the curly brackets is already there when the file is opened. We are simply telling it that if the list does not contain a joint in the list NoJoints, then the program should draw it.

You can also add and remove bones to connect the joints. This is as simple as commenting out the bones you don't want and following the code written for the bones to draw in new ones. The guide is:
this. DrawBone (skeleton.Joints, JointType.Joint1, JointType.Joint2);

Step 3: Step 2: Finding Joint Coordinates for Comparison

Picture of Step 2: Finding Joint Coordinates for Comparison

The main purpose of this program is to compare certain sets of joints and deciding if they are in optimal position. to do this we have to be able to track the position of the joints on the screen. to do this, we use mapMethod(j.Position).

first, we have to load the font that we will use for several functions throughout the program. scroll down till you see protected override void LoadContent (). In the brackets write this code:
this.font = Game.Content.Load<SpriteFont>("Segoe16");

Right underneath where we wrote the if statement for drawing certain joints, we will tell the program to track the joints on screen:

this.SharedSpriteBatch.DrawString(font, "(" + mapMethod(j.Position).Y + ")", mapMethod(j.Position), Color.Black);

the sprite batch function draws text on the screen and the mapMethod(j.Position) finds the joint position.
I only chose to write the Y coordinate on the screen because the X coordinate stays constant.

Step 4: Step 3: Comparing Joint Sets

Picture of Step 3: Comparing Joint Sets

This is the meaty part of the code writing!!
We chose to analyze four sets of joints: shoulders, hips, wrists, left knee and right ankle, and right knee and left ankle.
the goal was to have some sort of indication when one joint was out of placement. Out of placement in this scenario would be when the two joints were no longer level.

We wrote 2 functions to check levelness:
function1-- for shoulders, hips, and wrists (this can be added at the bottom of the skeleton stream renderer)
//functions We Added
        //used to check levelness of hips, wrists and shoulders
        private List<JointType> IsLevel(JointType j1, JointType j2)
        {
            List<JointType> higherJoints = new List<JointType>();
            foreach (var skeleton in skeletonData)
            {
                int Y1 = (int)mapMethod(skeleton.Joints[j1].Position).Y;
                int Y2 = (int)mapMethod(skeleton.Joints[j2].Position).Y;
                if (Math.Abs(Y1 - Y2) < 10)
                {
                    higherJoints.Add(JointType.Head); //returns the head joint because that joint is void
                }
                else
                {
                    if (Y2 < Y1) higherJoints.Add(j2);
                    if (Y1 < Y2) higherJoints.Add(j1);
                }
            }
            return higherJoints;
        }



Essentially, if the joints are level, it will return the Head Joint (which is nonexistent in the program) and then will be seen as green. However, if the distance between the joints is greater than ten, it turns the higher joint red, alerting the player that that particular joint is out of alignment.

Function 2 (Checks the knees and ankles):
//used to check placement of foot and knee in passe
        private List<JointType> IsHigher(JointType j3, JointType j4)
        {
            List<JointType> lowerJoints = new List<JointType>();
            foreach (var skeleton in skeletonData)
            {
                int Y3 = (int)mapMethod(skeleton.Joints[j3].Position).Y;
                int Y4 = (int)mapMethod(skeleton.Joints[j4].Position).Y;
                if (Math.Abs(Y3 - Y4) < 10 || Math.Abs(Y3-Y4) > 70 || Y3 < Y4)
                {
                    lowerJoints.Add(JointType.Head);
                }
                else
                {
                    lowerJoints.Add(j3);
                }
            }
            return lowerJoints;
        }


this does the same thing as function 1, but in the opposite direction. If the ankle joint is too far below the knee, it is colored red. All margins of error can be easily adjusted for different levels of dancers.

We then have to add these functions in as variables. Add the following code right before the bones are drawn:
//Variables Kelcy Added

            List<JointType> shoulderCheck = IsLevel(JointType.ShoulderRight, JointType.ShoulderLeft);
            List<JointType> hipCheck = IsLevel(JointType.HipRight, JointType.HipLeft);
            List<JointType> wristCheck = IsLevel(JointType.WristRight, JointType.WristLeft);
            List<JointType> kneeAndAnkleCheckR = IsHigher(JointType.AnkleRight, JointType.KneeLeft);
            List<JointType> kneeAndAnkleCheckL = IsHigher(JointType.AnkleLeft, JointType.KneeRight);

Step 5: Step 3 Continued:

Picture of Step 3 Continued:

We then have to relay the signals to the screen and give the program scenarios of joint levelness:

underneath the variables we added, we must tell it what to write in certain situations. We called them danceMessage and danceMessage2. Here we will also give the program our current skeleton that we want it to check:
             int currentSkeleton = 0;
            string danceMessage = "Lower your:";
            string danceMessage2 = "Raise your:";

We will use these dance messagesto give the player corrections as he/she dances:
this code will go underneath where the coordinates for the joints are printed

this.SharedSpriteBatch.DrawString(font, danceMessage, new, vector2(0,450), Color.Red);
this.SharedSpriteBatch.DrawString(font, danceMessage2, new, vector2(0,425), Color.Red);


This code will go in the section "now draw the joints" to check each set of joints:

if (shoulderCheck[currentSkeleton] != JointType.Head)
                            {
                                switch (shoulderCheck[currentSkeleton])
                                {
                                    case JointType.ShoulderLeft:
                                        danceMessage += "left shoulder, ";
                                        break;
                                    case JointType.ShoulderRight:
                                        danceMessage += "right shoulder, ";
                                        break;
                                }
                                if (j.JointType == shoulderCheck[currentSkeleton])
                                    jointColor = Color.Red;
                            }


This is for the shoulder joints and it tells that if one joint is higher, it will be colored red, indicating that it is out of alignment. Once it placed level again, it will become green. Now write the code for the hip and wrist joints as well!!

The knees and ankles have a slight adjustment and will only return one possibility. This is used to check the position of the dancers foot in passe (used for turning):

//Checks the right foot and left knee
                            if (kneeAndAnkleCheckR[currentSkeleton] != JointType.Head)
                            {
                                switch (kneeAndAnkleCheckR[currentSkeleton])
                                {
                                    case JointType.AnkleRight:
                                        danceMessage2 += "Right Foot, ";
                                        break;


Then write the coding for the left ankle and right knee!

I think that's about it!! There is a lot of cool stuff to mess around with in the program. You can change colors and positions of objects on the screen! GET CRAZY!!

Comments

garkuna (author)2016-06-09

Hello, i tried to download the code by the link posted on this 'ible, but it is broken by now, is there a chance you still upload it? pLSSSS

kelcynewton (author)garkuna2016-06-09

Did you mean the link to the Microsoft Kinect SDK's? If so, I just updated the link to what I believe is the most updated SDK Microsoft offers. I apologize for the broken link. I posted this project years ago, and haven't really looked at it since. Hope it helps!

stushapiro (author)2013-08-20

Is there a way to download the code to play with?

Youripasted (author)2012-08-29

Brilliant, Kelcy.

I have a Kinect and currently I'm at my final year in the university. I feel comfortable using Visual Studio and this is a great idea for further development. Thank you very much for the idea, and I think I'll borrow it for my final project.

About This Instructable

9,839views

61favorites

License:

Bio: Not all who wander are lost
More by kelcynewton:Using Kinect Hacks for Dance Training
Add instructable to: