Introduction: Virtual Reality Games and Making Your Body a Controller! (Unity + Visual Studio + Kinect + Oculus)
Immersive Infinite Runner
Go ahead and get started.
Supplies
· Microsoft Account – https://www.microsoft.com/en-us/account/default.aspx
· Visual Studio Community Edition 2013 or 2015 – https://www.visualstudio.com/
· Visual Studio Unity Plugin – http://unityvs.com
· Kinect SDK - https://www.microsoft.com/en-us/kinectforwindows/develop/downloads-docs.aspx
· Kinect for Windows Hardware (make sure you have Kinect Adapter for Windows) – https://www.microsoft.com/en-us/kinectforwindows/purchase/default.aspx#tab=2
· Unity3d version 4 (aka Unity4.6.1 or higher) or Unity 3d version 5 (aka Unity5)
· Oculus SDK
· Oculus Runtime
· Oculus – Unity 4 Integration (this works in Unity5 as well)
· Computer with both usb 2.0 and usb 3.0 ports
**Mac users** you will not be able to work with the Kinect unless you have Bootcamp installed (or some other way to partition your hard drive) with a windows OS, sorry.
The Game
Okay so, the first thing you'll need is to have your game all set up and running, in Unity. If you've been following my blog (http://kaharri.com – Unity Gaming series) then you should have the bulk of the game running. If not you can grab all the code in my GitHub repo – http://github.com/katvharris. There you’ll find the fully functional infinite runner without the Kinect and Oculus portions, that I’m using to build off of.
Step 1: Unity and Visual Studio
Unity and Visual Studio
In order to edit the code you need to have Unity installed on your computer. I explain it below as well.
First you need to download the Visual Studio Plugin for Unity. Go to http://unitvs.com
Download the version of the plugin that matches the version of Visual Studio you have. Once downloaded extract all the files to a location you can easily remember.
Open the game in Unity. Go to Assets> Import Package> Custom Package. Navigate to the Unity Visual Studio package and select it.
IMAGE 1 ABOVE
After selecting it, Unity will prompt you to import the package. Click Import.
Now that you have imported this package once to Unity, you will not have to go looking for the package again. If you close Unity, and reopen it, you will notice that now if you go to Assets>Import Package there will be a new option called Visual Studio Tools.
Next we have to specify that we want to use the Visual Studio IDE instead of MonoDevelop.
To do this we navigate Edit>Preferences
IMAGE 2 ABOVE
Go to External Settings
IMAGE 3 ABOVE
And Switch it from Mono Develop to Visual Studio. And there you have it! One small caveat. On each new project you create you still need to go to Assets>Import Package> Visual Studio Tools, and then confirm the import.
Step 2: Adding Virtual Reality
Oculus
Cool. So now that you have the game the virtual reality part will be provided by the Oculus. There are some other headsets but currently Oculus SDK has the most documentation and therefore if you run into an error it will be easier to find help online.
First, we need to grab the Oculus pieces from their site. You can go to their download section. Now if you have a Mac or Linux, download the appropriate links for those operating systems.
IMAGE 1
After you download the links you now need to install the runtime. Then restart your computer.
Integrating into Unity
In the project create a new scene to test the Oculus Piece.
IMAGE 2
Extract the files from the Unity Integration Package you downloaded. Go Into Unity to Assets>Import Package>Custom Package
Find where you extracted the files and navigate to the Unity Plugin.
IMAGE 3
Then double click on the package. Once you do, Unity will pull up all the files in the package and prompt you to Import them. Hit Import.
IMAGE 4
Now you should have a new folder in your Assets called OVR.
IMAGE 5
This new folder will have all the files that you’ll need to add virtual reality to the game.
Step 3: Using and Scripting Oculus Camera
Now using the Oculus Package is super easy. Oculus has already created prefabs for developers to use. They have a prefab with just the camera rig as well as one with the rig and a character motor.
IMAGE 1
To use them just Click and Drag it into your scene. I placed the OVRPlayerController at 0, 2, 0.
IMAGE 2
Now try running the game. You should have something that looks like this:
IMAGE 3
The double circle screen is what will be fed to your Oculus, and with the lenses and the headset it should become one image with a 3 dimensional feel. Normally this player prefab works for most games but for reasons unknown the collider and rigid body of the OVR prefab does not work with the game. So we have one more step. Combining the FPS Controller with the OVR Controller.
Creating OVR Player Prefab
Now in the scene add the FPSprefab in the project by searching for it in the assets folder.
IMAGE 4
Click and drag it into the scene like you did with the OVRPlayer Prefab and place it at (0, 0, 0).
IMAGE 5
Next Drag the OVRCamera Prefab into the scene under the Head Joint of the FPSPreFab.
IMAGE 6
IMAGE 7
After that disable the First Person Camera.
IMAGE 8
The OVRCamera Rig is perfect for this situation since we already have a working player we want to use. Move the FPSPreFab to (0, 2, 0) and Run the game. It should look like this: (And the arrow keys or “wasd” will still control movement and mouse/head will control where you look).
IMAGE 9
Great now that we know it works, we need to add it to the main scene. So go to File>Open Scene. It should promt you to save this scene as well. Save this scene and call it OVRTest since we made this enviroment to test the OVR libraries.
In the Open Scene Explorer window go to the Scenes folder> then select Main
IMAGE 10
Step 4: Creating VR UI
Now in our main scene, add the OVRCamerRig to the FPSPreFab like we did before. We now need to add a UI for the Oculus because the oculus cannot use HUD displays the same way regular Unity Cameras can. So to do this copy the Canvas object and paste it under the OVRCameraRig and rename the Canvas OVRCanvas.
IMAGE 1 ABOVE
IMAGE 2 ABOVE
We will have to edit this Canvas a bit before it works.
1) We need this UI to be in World Space. We do this by selecting the OVRCanvas and then Unity under Canvas>RenderMode and in the pull down select World space.
IMAGE 3 ABOVE
IMAGE 4 ABOVE
2) Change the Instructions Text
IMAGE 5
IMAGE 6
3) Disable the Buttons – Buttons won’t be needed since we are using the Kinect.
4) Change the Animation in the GameController to apply to the new OVRCanvas – Do this by adding these lines to the GameControllers awake method:
GameObject ovrcanvas = GameObject.Find("OVRCanvas");startAnim = ovrcanvas.GetComponent();
5) And comment out the current startAnim line.
//startAnim = canvas.GetComponent();
We aren’t done yet. We also need to readjust the End Game UI but we can achieve this the same way we did it with the StartUI.
Select the Start UI from the hierarchy and select the 2D button in the Scene Tab. In general it’s easier to work with UI in the 2D mode of Unity.
IMAGE 7 ABOVE
IMAGE 8 ABOVE
We don’t really see anything in the scene. That’s because at the start of the game the only thing in the canvas that we want to see are the instructions. However, we do see a rectangle with blue circles on the corners surrounding the Canvas. That rectangle signifies the space where the End UI will be shown. Now in Oculus the view finder is a bit limited since we have the canvas in world space. Therefore we need to move the contents of the UI to the viewing area of the users vision; which happens to be the instructions box.
To move it we need to make sure that the anchor movement tool is selected and not the regular transition tool.
IMAGE 9
IMAGE 10
Then we can move around the contents towards the center of the screen. By clicking and dragging. When you have certain parts of the hierarchy selected their corresponding positions in the scene will have the blue cornered rectangle around them.
Start Button UI
In the GameController’s Update method we need to add a conditional that qualifies the game has not started and the animation is not loading. Conisidentaly we have this already. In the Boolean variable gamestarted.
Add this to the beginning of the Update.
if (!gamestarted)
{
if (Input.GetKey(KeyCode.KeypadEnter))
{
StartGame();
}
}
The Update should now look like this:
void Update ()
{
if (!gamestarted)
{
if (Input.GetKey(KeyCode.KeypadEnter))
{
StartGame();
}
}
if (loading)
{
Debug.Log("Anime should start");
// ... tell the animator the game is over.
startAnim.SetTrigger("StartGame");
// .. increment a timer to count up to restarting.
beginningTimer += Time.deltaTime;
// .. if it reaches the restart delay...
if (beginningTimer >= beginningDelay)
{
Debug.Log("TriggeringStart");
loading = false;
StartGame();
}
}
// Set the displayed text to be the word "Score" followed by the score value.
scoreText.text = "Score: " + score;
finalScoreText.text = "Score:" + score;
if (playerHealth.currentHealth <= 0 && !gameended)
InitializeEndGame();
if (gameended)
{
restartTimer += Time.deltaTime;
// .. if it reaches the restart delay...
if (restartTimer >= restartDelay)
{
// .. then reload the currently loaded level.
Application.LoadLevel(Application.loadedLevel);
}
}
}
The next step we need to complete is to modify the Game Controller, so it can end and restart the game with our new inputs. We will also need to destroy the platforms of the scene, otherwise it will block our players view like the picture below.
IMAGE 11
On the left is what the players view should see, but instead they still see the platforms.
Step 5: Adding EndGame UI and Controls
Currently all the platforms in the Game are children on the Problem Solver game object. This is how all the rotations can be applied to even the newly spawned platforms. So we can easily deactivate them by deactivating the ProblemSolver.
IMAGE 1
In the Game Controller’s Update method there is this section of code at the bottom:
if (playerHealth.currentHealth <= 0 && !gameended)InitializeEndGame();
if (gameended)
{ restartTimer += Time.deltaTime; // .. if it reaches the restart delay... if (restartTimer >= restartDelay) { // .. then reload the currently loaded level. Application.LoadLevel(Application.loadedLevel); } }
What this does is display the Game End animation canvas, which is already created, and after a specified amount of time the game restarts. Now when in the IntializeEndGame method add the following line:
problemSolverObject.SetActive(false);
So the method now looks like this:
void InitializeEndGame()
{
startAnim.SetTrigger("EndGame");
player.GetComponent().enabled = false;
player.rigidbody.useGravity = false;
//Remove Platforms
problemSolverObject.SetActive(false);
}Now when the game ends we can see the End Screen.
IMAGE 2 ABOVE
Okay so all that’s left to do make a key board trigger for the restart and change the UI of the EndGameCanavas to inform the user how to restart the game.
To change the text repeat what we did with the StartUI.
IMAGE 3 ABOVE
IMAGE 4 ABOVE
The code behind for restarting the game happens in the Update method of the GameController. Add this under the player health conditional.
if (playerHealth.currentHealth <= 0 && !gameended)
InitializeEndGame();
if (Input.GetKeyUp(KeyCode.Backspace) && !gameended)
RestartGame();
So run the game again, and when you die you should see the white game over screen with the new text and should restart when you hit Backspace.
IMAGE 6 ABOVE
Finally we are done with the Oculus component. Now to implement for those Kinect controls!
Step 6: Setting Up the Kinect
Software Setup
Now before you start just plugging in any Kinect from random Xbox One or Xbox 360, you need the appropriate software so your computer knows what you’re plugging into it. You also can’t use any rando Kinect. This tutorial uses the Kinect v2; that’s the one that is shipped with the Xbox One. However, in order to use the v2 with your computer you need to make sure you have the Windows adapter!
First let’s install the SDK. We need to go here:
https://www.microsoft.com/en-us/kinectforwindows/d...
You can also go to the Kinect for Windows main site and go to their Technical documentation and tools. On the Technical Documentation and Tools page they have a list of useful links for documentation and essential downloads, including the Unity Plugin that we will also need.
IMAGE 1
After you download the SDK, run it. You will be prompted with a Kinect for Windows Setup wizard.
IMAGE 2
After you install the SDK you should have some new programs installed on your computer.
· SDK Browser
· Kinect Studio v 2.0
· Kinect Gesture Builder
IMAGE 3
The SDK Browser will be the most useful of the new software, because it contains links to all the other programs/software as well as demos and example code you can use.
Hardware Setup
Setting up the hardware is pretty straight forward, which is great!
**Mac users** you will not be able to work with the Kinect unless you have Bootcamp installed (or some other way to partition your hard drive) with a windows OS, sorry.
IMAGE 4
If you need more help setting up the hardware here is a helpful guide from Microsoft: https://www.microsoft.com/en-us/kinectforwindows/...
Once you Plug in the Kinect to the USB3 port on your computer we can test the connection with the Kinect SDK applications that were automatically downloaded.
Open the SDK Browser for Kinect. The first program in the list should be the Kinect Configuration Verifier. Run it. The Kinect’s light should now turn on if it’s connected properly and the Verifier will open a window, if everything is correct, it should look like this:
IMAGE 5
If something is wrong you will get a red X identifying the error:
IMAGE 6
Even though I have a warning about my USB port, my Kinect still runs. Now to see the information collected by the Kinect we will run Kinect Studio. It will be the third item in the SDK browser; or you can open it straight from your applications.
Kinect Studio looks like this:
IMAGE 7
Don’t worry if you don’t see anything. At startup. Although your Kinect is on and streaming data to your computer Kinect Studio isn’t reading in that data yet.
You need to hit the plug icon at the top left hand corner so you can see anything.
IMAGE 8
IMAGE 9
There are 7 streams of data coming from the Kinect:
1) Body Frame
2) Body index
3) Calibration Data
4) Depth
5) IR
6) Title Audio
7) Uncompressed Color
You can toggle what streams of information you want to receive by deselecting the check boxes on the left. The most important streams of data are the Body Frame/Body Index, Depth, and Infra Red (IR).
You can close Kinect Studio we don’t need it anymore.
Kinect and Unity
First step is download the Kinect for Windows Unity Package. You can find it at this link: https://www.microsoft.com/en-us/kinectforwindows/...
IMAGE 10
To get started we will use the steps directly outlined in the Kinect for Windows Unity Package from Microsoft; but slightly edited since we already have a project created.
1) Expand the .Zip file, and move Kinect.2.0.1410.19000.UnityPackageto a well known
2) Open UnityPro (you need to have a Pro edition to pull in custom packages and plugins)
3) Open your Unity Project
4) Click on the menu item Assets->Import Package->Custom Package...
5) Navigate to the from step 1
6) Select the Kinect.2.0.1410.19000.UnityPackage
7) Click "Open"
**Before you do step 8 here is an important thing to note – When Importing notice that the folder is called StandardAssets. This is the same name as the Sample Assets from the Unity Store. If you are using Unity’s Standard Assets package the import manager will embed the new Kinect files into the already existing folder. Be careful! If you don’t keep track of what you’re importing you might lose files within the numerous folders of your project. So, to keep things organized keep note of the files that are not in subfolders and are just in the Standard Assets folder.
In this case there are 10:
· EventPump.cs
· KinectBuffer.cs
· KinectSpecialCases.cs
· CameraIntrinsics Import Settings
· CollectionMap Import Settings
· ExecptionHelper Import Settings
· INativeWrapper Import Settings
· NativeObject Import Settings
· SmartGCHandle Import Settings
· ThreadSafeDictionary Import Settings
Okay now back to the steps**
8) Click "Import" in the lower right hand corner of the "Importing Package" Dialog (which Unity will launch after step 7)
9) If you wish to see the Kinect in action there are two sample scenes available from the zip.
10) If you wish to use VisualGestureBuilder within Unity, repeat steps 1 through 8 with Kinect.VisualGestureBuilder.2.0.1410.19000.unitypackage
11) If you wish to use the Face functionality within Unity, repeat steps 1 through 8 with Kinect.Face.2.0.1410.19000.unitypackage
Okay lets stay organized. Create a new Folder in your Assets Folder called KinectPackage. Then, add the 10 files I mentioned above as well as Windows Folder from StandardAssets.
IMAGE 11 ABOVE
Step 7: Kinect Scripting
Create a new Folder inside your Scripts folder Called KinectScripts. Now in that folder create two new scripts; one called BodyManger and the other called BodyView.
IMAGE 1 ABOVE
BodyManager
First we need BodyManager - this object will Mange the Kinect sensor connection and read in all the body data coming from the Kinect.
Import the Kinect Library with:
using Windows.Kinect;
Then we need some fields for the manager to store and the data and get sensor data.
private KinectSensor _Sensor;
private BodyFrameReader _Reader;
private Body[] _Data = null;
public Body[] GetData()
{
return _Data;
}
Okay so now in the Start method we want to establish the connection for the Kinect.
void Start()
{
_Sensor = KinectSensor.GetDefault();
if (_Sensor != null)
{
_Reader = _Sensor.BodyFrameSource.OpenReader();
if (!_Sensor.IsOpen)
{
_Sensor.Open();
}
}
}
Now that the connection is open and reading in the data we need to store it in the Body array. We will do this every frame of the game, therefore we need to edit the Update() method.
First we check to see if the _Reader has been established and the connection has been completed. If it has we will take the last frame, the reader read in and if it’s not null, we can then check to see if the data is there.
void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if(frame = != null)
{
if(_Data == null)
{
//Write Code here
}
}
}
}
Now we need to get the Body data from the Senor. To do this we will need to create a new Body array with data from the Sensor.BodyFrameSource.BodyCount.
At the end the method should look like this:
void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
if (_Data == null)
{
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}
}
}
}
Then we need to refresh the stream of data from the Reader. By adding the following code to manipulate the frame.
void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
if (_Data == null)
{
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}
frame.GetAndRefreshBodyData(_Data);
frame.Dispose();
frame = null;
}
}
}
The last method in the Body Manager Class is OnApplicationQuit(), which Disposes the Reader, and closes the Sensor stream, sets it to null.
void OnApplicationQuit()
{
if (_Reader != null)
{
_Reader.Dispose();
_Reader = null;
}
if (_Sensor != null)
{
if (_Sensor.IsOpen)
{
_Sensor.Close();
}
_Sensor = null;
}
}
Body View
The next Script to write is one to draw the skeletal structure. We won’t necessarily need to see the skeleton for the game, however, I’ll show you how to show skeletal body tracking. We also need the skeletal data to track the hands, whose state will dictate controller commands.
For this MonoBehavoir class we will need, a material to draw the bone in the Unity scene. A gameobject to store the BodyManger, to control the stream of the Kinect.
public Material BoneMaterial; public GameObject BodyManager;
We also need a BodyManager object and a Dictionary to store the bodies being tracked.
private Dictionary _Bodies = new Dictionary();
private BodyManager _BodyManager;
Next we need to map out all the bones by the two joints that they will be connected to.
private Dictionary _BoneMap = new Dictionary()
{
{ Kinect.JointType.FootLeft, Kinect.JointType.AnkleLeft },
{ Kinect.JointType.AnkleLeft, Kinect.JointType.KneeLeft },
{ Kinect.JointType.KneeLeft, Kinect.JointType.HipLeft },
{ Kinect.JointType.HipLeft, Kinect.JointType.SpineBase },
{ Kinect.JointType.FootRight, Kinect.JointType.AnkleRight },
{ Kinect.JointType.AnkleRight, Kinect.JointType.KneeRight },
{ Kinect.JointType.KneeRight, Kinect.JointType.HipRight },
{ Kinect.JointType.HipRight, Kinect.JointType.SpineBase },
{ Kinect.JointType.HandTipLeft, Kinect.JointType.HandLeft }, //Need this for HandSates
{ Kinect.JointType.ThumbLeft, Kinect.JointType.HandLeft },
{ Kinect.JointType.HandLeft, Kinect.JointType.WristLeft },
{ Kinect.JointType.WristLeft, Kinect.JointType.ElbowLeft },
{ Kinect.JointType.ElbowLeft, Kinect.JointType.ShoulderLeft },
{ Kinect.JointType.ShoulderLeft, Kinect.JointType.SpineShoulder },
{ Kinect.JointType.HandTipRight, Kinect.JointType.HandRight }, //Needthis for Hand State
{ Kinect.JointType.ThumbRight, Kinect.JointType.HandRight },
{ Kinect.JointType.HandRight, Kinect.JointType.WristRight },
{ Kinect.JointType.WristRight, Kinect.JointType.ElbowRight },
{ Kinect.JointType.ElbowRight, Kinect.JointType.ShoulderRight },
{ Kinect.JointType.ShoulderRight, Kinect.JointType.SpineShoulder },
{ Kinect.JointType.SpineBase, Kinect.JointType.SpineMid },
{ Kinect.JointType.SpineMid, Kinect.JointType.SpineShoulder },
{ Kinect.JointType.SpineShoulder, Kinect.JointType.Neck },
{ Kinect.JointType.Neck, Kinect.JointType.Head },
};
BodyView Update()
Now in the Unity Update() method we need to check to see if the Body Manager is not null and that it has data.
void Update()
{
int state = 0;
if (BodyManager == null)
{
return;
}
_BodyManager = BodyManager.GetComponent();
if (_BodyManager == null)
{
return
}
Kinect.Body[] data = _BodyManager.GetData();
if (data == null){
return;
}
}
Next, while still in the Update() method, we need to get the amount of bodies in the list of tracked bodies. Then delete unknown bodies.
List trackedIds = new List ();
foreach (var body in data)
{
if (body == null)
{
continue;
}
if (body.IsTracked)
{
trackedIds.Add(body.TrackingId);
}
}
List knownIds = new List (_Bodies.Keys);
// First delete untracked bodies
foreach (ulong trackingId in knownIds)
{
if (!trackedIds.Contains(trackingId))
{
Destroy(_Bodies[trackingId]);
_Bodies.Remove(trackingId);
}
}
Now that we have the keys for tracking the bodies we need to create a body object with that tracking ID key. We need to write two more methods. A CreateBodyObject() method that will take a ulong id and a RefreashBodyObject() method that will take a Kinect.Body object and a GameObject for the body. We will use these methods after we go through the data, and find if bodies inside are being tracked or not. If it is tracked but doesn’t have a TrackingId, then we need to create a body with that TrackingID. If it is being tracked and has a TrackingId then we just need to refresh the drawn body.
foreach (var body in data)
{
if (body == null)
{
continue;
}
if (body.IsTracked)
{
if (!_Bodies.ContainsKey(body.TrackingId))
{
_Bodies[body.TrackingId] = CreateBodyObject(body.TrackingId);
}
RefreshBodyObject(body, _Bodies[body.TrackingId]);
}
}
}
CreateBodyObject()
The CreateBodyObject takes an ID and returns a body gameobject. So we first need to create a gameobject that will store the appropriate data retrieved; then we need a for loop to go through every joint to draw the body.
private GameObject CreateBodyObject(ulong id)
{
GameObject body = new GameObject("Body:" + id);
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++){
}
return body;
}
For every joint in the body we create a cube and add a lineRenderer to that cube. The cube will be drawn at each joint while the line renderer will be drawn to connect the joints.
private GameObject CreateBodyObject(ulong id) {
GameObject body = new GameObject("Body:" + id);
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++){
GameObject jointObj = GameObject.CreatePrimitive(PrimitiveType.Cube);
LineRenderer lr = jointObj.AddComponent();
lr.SetVertexCount(2);
lr.material = BoneMaterial;
lr.SetWidth(0.05f, 0.05f);
jointObj.transform.localScale = new Vector3(0.3f, 0.3f, 0.3f);
jointObj.name = jt.ToString();
jointObj.transform.parent = body.transform;
}
return body;
}RefreashBodyObject()
Now to write the ResfreshBodyObject method. In this method we need to go through each joint type possible just like we did in the CreateBodyObject method. But this time we are passing in the current body, as well as the appropriate tracking number so we don’t draw the bones for the wrong person.
private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
{
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
{
}
}
Inside this for loop we need to get the key value pairs we made before in the bone loop for each joint.
private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
{
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
{
Kinect.Joint sourceJoint = body.Joints[jt];
Kinect.Joint? targetJoint = null;
if(_BoneMap.ContainsKey(jt))
{
targetJoint = body.Joints[_BoneMap[jt]];
}
}
}
We also need to update the skeletons position so it’s in the accurate place on the screen. To do this we need to write a method to get the Vetcor3 from the sourceJoint.
private static Vector3 GetVector3FromJoint(Kinect.Joint joint)
{
return new Vector3(joint.Position.X * 10, joint.Position.Y * 10, joint.Position.Z * 10);
}
The scale by 10 is to enlarge the skeleton, which will make it easier to work with. Now we have position to correct the gameObjects position.
private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
{
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
{
Kinect.Joint sourceJoint = body.Joints[jt];
Kinect.Joint? targetJoint = null;
if(_BoneMap.ContainsKey(jt)){
targetJoint = body.Joints[_BoneMap[jt]];
}
Transform jointObj = bodyObject.transform.FindChild(jt.ToString());
jointObj.localPosition = GetVector3FromJoint(sourceJoint);
}
}
Next step in the for loop is to get the linerenderer from the bodyObject, which was the cube we create for each joint. Then we need to see if target joint has a value. If it does we can then draw a line from the original joint to the target.
LineRenderer lr = jointObj.GetComponent ();
if(targetJoint.HasValue)
{
lr.SetPosition(0, jointObj.localPosition);
lr.SetPosition(1, GetVector3FromJoint(targetJoint.Value));
}
else
{
lr.enabled = false;
}
Great! So we are almost done with drawing the skeleton. There is a bit more information that will be helpful that the SDK gives you, which is tracking status. There are three states to choose from, Tracked, Inferred, or NotTracked. We can have the line renderer show us the state of tracking by changing it’s color. To do this we need a method that will return a color based on the current state.
private static Color GetColorForState(Kinect.TrackingState state)
{
switch (state){
case Kinect.TrackingState.Tracked:
return Color.green;
case Kinect.TrackingState.Inferred:
return Color.red;
default:
return Color.black;
}
}
Now we add one more line to the for loop of the RefreachBodyObject method and we are done.
private void RefreshBodyObject(Kinect.Body body, GameObject bodyObject)
{
for (Kinect.JointType jt = Kinect.JointType.SpineBase; jt <= Kinect.JointType.ThumbRight; jt++)
{
Kinect.Joint sourceJoint = body.Joints[jt];
Kinect.Joint? targetJoint = null;
if (_BoneMap.ContainsKey(jt))
{
targetJoint = body.Joints[_BoneMap[jt]];
}
Transform jointObj = bodyObject.transform.FindChild(jt.ToString());
jointObj.localPosition = GetVector3FromJoint(sourceJoint);
LineRenderer lr = jointObj.GetComponent();
if (targetJoint.HasValue)
{
lr.SetPosition(0, jointObj.localPosition);
lr.SetPosition(1, GetVector3FromJoint(targetJoint.Value));
lr.SetColors(GetColorForState(sourceJoint.TrackingState), GetColorForState(targetJoint.Value.TrackingState));
}
else
{
lr.enabled = false;
}
}
}
And that’s it for drawing the skeleton!
Step 8: Adding Kinect to Unity
Now in Unity I’ve created another scene by going to file new scene (it will prompt you to save your current scene if you haven’t already). This empty scene will make it easier for you to test and see what you’re doing.
In the empty scene, which I have saved as kinectTest, create two empty game objects. Call them Body Manager and Body View.
IMAGE 1
Attach the Body Manager script to the body manager object. Attach the Body View script to the BodyView object.
IMAGE 2
Select the Body View object. In the inspector you need to fill in the Body Manager slot and Bone Material slot. Click and drag the BodyManager from the hierarchy to the slot. Then in your materials folder click and drag the Bone Material into the bone material slot.
IMAGE 3
Now hit run. You should see nothing on your screen at first. This is because the Kinect hasn’t registered your skeleton yet. Stand back about 4 ft, and you should see the skeleton appear.
IMAGE 4
If you look closely you can tell that the legs of the skeleton are red. This is because the Kinect is interpreting that’s where my legs should be. My desk was blocking the sensor from detecting where my legs actually were.
But wait! You say. We don’t need a skeleton in our game, just the hand gestures and states! So was all this a waste? NO! Of course it wasn’t. We just don’t need to draw the whole skeleton in the scene anymore. But we now have the know how if you need to in the future. YAY!
Step 9: Hand Controls With the Kinect
The next script we will create will be the one we use in our game along with the body manager script. It will detect the hand state and trigger the rotation for our game.
Hand Manager Script
Now create another script in the same folder as the body manager script and body view manager script. Call it HandManager. This script will look a lot like our Body View controller.
Add the following from BodyView to our HandManager Script.
using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using Kinect = Windows.Kinect;
public class HandManager : MonoBehaviour
{
public GameObject BodyManager;
private PlatformRotate platformRotate;
private OVRInfinitePlayerController ovrPlayerScript;
private InfinitePlayerController playerScript;
private Dictionary _Bodies = new Dictionary ();
private BodyManager _BodyManager;
private float flipTimer;
private float flipDelay = 2;
int prevState = 0;
private Dictionary _BoneMap = new Dictionary()
{
{ Kinect.JointType.FootLeft, Kinect.JointType.AnkleLeft },
{ Kinect.JointType.AnkleLeft, Kinect.JointType.KneeLeft },
{ Kinect.JointType.KneeLeft, Kinect.JointType.HipLeft },
{ Kinect.JointType.HipLeft, Kinect.JointType.SpineBase },
{ Kinect.JointType.FootRight, Kinect.JointType.AnkleRight },
{ Kinect.JointType.AnkleRight, Kinect.JointType.KneeRight },
{ Kinect.JointType.KneeRight, Kinect.JointType.HipRight },
{ Kinect.JointType.HipRight, Kinect.JointType.SpineBase },
{ Kinect.JointType.HandTipLeft, Kinect.JointType.HandLeft }, //Need this for HandSates
{ Kinect.JointType.ThumbLeft, Kinect.JointType.HandLeft },
{ Kinect.JointType.HandLeft, Kinect.JointType.WristLeft },
{ Kinect.JointType.WristLeft, Kinect.JointType.ElbowLeft },
{ Kinect.JointType.ElbowLeft, Kinect.JointType.ShoulderLeft },
{ Kinect.JointType.ShoulderLeft, Kinect.JointType.SpineShoulder },
{ Kinect.JointType.HandTipRight, Kinect.JointType.HandRight }, //Needthis for Hand State
{ Kinect.JointType.ThumbRight, Kinect.JointType.HandRight },
{ Kinect.JointType.HandRight, Kinect.JointType.WristRight },
{ Kinect.JointType.WristRight, Kinect.JointType.ElbowRight },
{ Kinect.JointType.ElbowRight, Kinect.JointType.ShoulderRight },
{ Kinect.JointType.ShoulderRight, Kinect.JointType.SpineShoulder },
{ Kinect.JointType.SpineBase, Kinect.JointType.SpineMid },
{ Kinect.JointType.SpineMid, Kinect.JointType.SpineShoulder },
{ Kinect.JointType.SpineShoulder, Kinect.JointType.Neck },
{ Kinect.JointType.Neck, Kinect.JointType.Head },
};
Now we added a variable that’s not in our bodyview script which is the prevState variable.
int prevState = 0;
We have this variable to prevent constant rotating. In the regular input controls of the game we do this by only activating rotation when the key is that was pressed, goes up. With the Kinect it’s good to have a rest state that must be achieved if we want to rotate or flip again. In our case the rest state is any hand state that does not signal a rotation; for example, when both hands are in fists.
Now in the Update method we need to add the following:
void Update()
{
int state = 0;
if (BodyManager == null)
{
return;
}
_BodyManager = BodyManager.GetComponent();
if (_BodyManager == null)
{
return;
}
Kinect.Body[] data = _BodyManager.GetData();
if (data == null)
{
return;
}
List<ulong> trackedIds = new List<ulong>();
foreach (var body in data)
{
if (body == null)
{
continue;
}
if (body.IsTracked)
{
trackedIds.Add(body.TrackingId);
}
}
List<ulong> knownIds = new List<ulong>(_Bodies.Keys);
// First delete untracked bodies
foreach (ulong trackingId in knownIds)
{
if (!trackedIds.Contains(trackingId))
{
Destroy(_Bodies[trackingId]);
_Bodies.Remove(trackingId);
}
}
}
All the code above is the same as Body View, which is helpful since we already know how it works.
The only thing that is different is:
int state = 0;
This variable will hold the current hand state, and at the end of the Update method, we will save it in the previous state variable.
The next part of the update method that is different is in the foreach loop.
void Update()
{
int state = 0;
if (BodyManager == null)
{
return;
}
_BodyManager = BodyManager.GetComponent ();
if (_BodyManager == null)
{
return;
}
Kinect.Body[] data = _BodyManager.GetData();
if (data == null)
{
return;
}
List trackedIds = new List ();
foreach (var body in data)
{
if (body == null)
{
continue;
}
if (body.IsTracked)
{
trackedIds.Add(body.TrackingId);
}
}
List knownIds = new List (_Bodies.Keys);
// First delete untracked bodies
foreach (ulong trackingId in knownIds)
{
if (!trackedIds.Contains(trackingId))
{
Destroy(_Bodies[trackingId]);
_Bodies.Remove(trackingId);
}
}
foreach (var body in data)
{
if (body == null)
{
continue;
}
if (body.IsTracked)
{
//Check Hand State if body is being Tracked
state = CheckLeftHandState(body.HandLeftState) + (2*CheckRightHandState(body.HandRightState));
}
}
}
We need to write the CheckLeftHandState and CheckRIghtHandState methods. These methods take the Kinect.HandState parameter that equates to an enumerable of Closed, Open, or Lasso. Both will return an int that we will set equal to the state variable to tell the game what command we want to implement.
private int CheckRightHandState(Kinect.HandState handState2)
{
int result = 0;
switch (handState2)
{
//Normal
case Kinect.HandState.Closed:
result = 0;
break;
//Stop
case Kinect.HandState.Open:
result = 0;
break;
//Flip
case Kinect.HandState.Lasso:
result = 1;
break;
default:
result = 0;
break;
}
return result;
}
private int CheckLeftHandState(Kinect.HandState handState1)
{
int result = 0;
switch (handState1)
{
//Normal
case Kinect.HandState.Closed:
result = 0;
break;
//Flip
case Kinect.HandState.Open:
result = 3;
break;
//Left
case Kinect.HandState.Lasso:
result = 1;
break;
default:
result = 0;
break;
}
return result;
}
Originally the game flipped if both hands were in the lasso state, but that’s harder to control, because for a frame one hand might be recognized slightly faster than the other, and thus the game assumes we are rotating instead of flipping.
So now the game: rotates right if the right hand alone is in lasso state; rotates left if the left hand alone is in lasso state; and flips if the left hand is open.
With both the CheckLeftHandState and CheckRightHandState methods done we can go back to the Update method and finish it off. Currently we just need to see if the Kinect registers our hand states properly. To do this we create a switch statement that uses the current state and then checks what the previous state was, preventing continuous rotation. If the previous state was the restful state we Rotate, or in this case print to the Debug screen.
if (body.IsTracked)
{
//Check Hand State if body is being Tracked
state = CheckLeftHandState(body.HandLeftState) + (2*CheckRightHandState(body.HandRightState));
switch (state)
{
case 0:
break;
case 1:
//Left
if (prevState == 0)
Debug.Log("Left");
break;
case 2:
//Right
if (prevState == 0)
Debug.Log("Right");
break;
case 3:
//Both
if (prevState == 0)
{
Debug.Log("Both");
}
break;
default:
break;
}
prevState = state;
}
}
}
Okay so let’s see if this works. In your scene create another empty game object and call it HandManager. Attach the handmanager script to it. Remember to drag the BodyManager from the hierarchy to the BodyManger placeholder in the HandManager Inspector.
IMAGE 1
Run the code in Unity.
IMAGE 2
Okay but we still see the skeleton. Well if we deactivate the bodyView by checking the left hand box in the inspector we can run the project again.
IMAGE 3
BAM! There it is, and you know it’s still working based on the Debugger even if you can’t see the skeleton anymore!
Okay now to add this into the game. First let’s make the prefabs out of the HandManager and BodyManager objects.
IMAGE 4
Once you’ve create them drag the Body Manager from the Hierarchy to the Body Manager Prefab you created. Do the same with the Hand Manager. Now as prefabs we can easily move those objects and functionality into our main game scene!
Step 10: Kinecting the Game Input Controller
See what I did there. Kinecting…like connecting…Jokes! Anyway, were on the last stretch of getting everything together!
Switch back to your main scene. Add the prefabs into the Hierarchy.
IMAGE 1
Now we have to do a few things before our hands can control the rotation and flipping of the world. First we have to link the HandManager to the Rotator object. Then we have to make sure we don’t accidentally rotate at the beginning of the game.
Add the following variables to the HandManager:
public GameObject Rotator;
private PlatformRotate platformRotate;
We make the Rotator object public so we can set it inside of Unity later. Then in the start method set the platformRotate to the Rotators PlatformRotate;
void Start()
{
platformRotate = Rotator.GetComponent();
}
Now that our platformRotate is linked to the Rotators PlatfromRotate we can use it to call certain methods like TurnRight() and TurnLeft(). So let’s implement that. Remember in our update method where we call the total state of what the hands are doing? Well that’s where we are going to modify the code.
if (body.IsTracked)
{
//Check Hand State if body is being Tracked
state = CheckLeftHandState(body.HandLeftState) + (2*CheckRightHandState(body.HandRightState));
switch (state)
{
case 0:
break;
case 1:
//Left
Debug.Log("Left");
if (prevState == 0)
platformRotate.TurnLeft();
break;
case 2:
//Right
Debug.Log("Right");
if (prevState == 0)
platformRotate.TurnRight();
break;
case 3:
//Both
Debug.Log("Both");
if (prevState == 0)
{
platformRotate.Flip();
}
break;
default:
break;
}
prevState = state;
}
}
Okay so now we are so close to the finish line!!
Now we need to make sure we don’t call Flip or Rotate too soon, like on the instructions page.
Step 11: Enabling/Disabling Controls
Now to prevent the Kinect from sensing us too soon and accidentally trying to flip or rotate pieces of the game we need to disable the HandManager until the Enter button has been pressed.
Disable/Enable Kinect controls on Load
To do this go into the GameController script. Add the following variables to store the HandManger object that is in the scene and the HandManagerScript
public GameObject handController;
Then at the bottom of the Awake method add this line:
handController.SetActive(false);
Next find the StartGame() method and add the following line to the else:
handController.SetActive(true);
It should now look like this:
public void StartGame()<br>{<br> if (startingPlay)<br> {<br> Debug.Log("Hit Button");<br> if (!gamestarted && !loading)<br> {<br> gamestarted = true;<br> loading = true;<br> Debug.Log("gameStarted and loading");<br> }<br> else<br> {<br> //loading complete, start true<br> Debug.Log("Spawning and Enabling Player");<br> handController.SetActive(true);<br> testSpawner.StartGeneration();<br> startingPlay = false;<br> }<br> }<br>}For those of you who are wondering about why we don’t also do this to do BodyManager, the answer is: We still want to establish connection as soon as the game starts.
AND WE ARE DONE!! Run the Game and look at the awesomeness that is INFINITE RUNNER WITH KINECT AND OCULUS INTEGRATION!!! All that's left is for you to publish it so your friends can playin too!
Happy Coding!!
-TheNappingKat
