Prototyping Interactive Environments in Virtual Reality With Google Cardboard, Unity and Hotline Bling (TfCD)

These days, (immersive) experiences are prioritised over pure functionality and specs on paper. This change of focus has influenced the design process of industrial designers, architects, app-developers and artists, striving to create prototypes of what a concept would feel like. Virtual reality is solution for simulating an interactive environment with great potential, through a fast iteration cycle and low investment.

Google has made virtual reality available to the masses with Cardboard, a virtual reality headset for your phone made of, you guessed it, cardboard. The screen of the phone, corrected through a lens for each eye, creates a stereoscopic image that fills almost your entire field of view. The gyroscope and accelerometer of the phone track the rotation of your head, so that you can look around in the virtual reality environment freely.

Unity is a free game engine that supports the production of games for almost any platform imaginable, from mobile to desktop to console. Google has build a SDK (software development kit) for using Cardboard in Unity, making it quite easy to get started.

In this tutorial

• Setting up Unity for VR with Cardboard
• Creating a simple VR scene, and loading it on your phone
• Moving around in the environment
• Creating atmosphere with lights, shaders and shadows (while keeping performance snappy)

• Interactivity with objects in the virtual reality scene

It is finally concluded with an evaluation of VR as a technique for prototyping concepts for the real world.

Step 1: What You Need Before You Get Started

Smartphone running iOS or Android

Take into account that in VR, your phone's screen is observed very closely. My phone has a 4.6" 1280x720 screen, and offers a satisfactory experience. Still, a larger phone with a Full-HD or WQHD screen will be more immersive and perhaps a tad bit sharper.

Google Cardboard (compatible) headset

Many alternatives are available, from cheap knock-offs to alternatives with sturdier materials (plastic or even metal). If you plan to use it repeatedly, it might be a good idea to invest 20-30 euro in a premium alternative. Also make sure you have a viewer that fits the size of the phone that you intend to use. This doesn't have to be precise, but usually a range of compatible screen-sizes is provided.

Unity

Download and install Unity from https://unity3d.com/. The free version is all that is necessary.

Google Cardboard SDK for Unity

Download the SDK from https://developers.google.com/cardboard/unity/download.

Mobile development

The additional tools needed to make a mobile app / game depend on the platform.

For Android, download the Android SDK Tools from http://developer.android.com/sdk/index.html#Other. Use them to install a recent SDK version (see http://developer.android.com/sdk/installing/index.html?pkg=tools for more info). Also, enable developer mode (go to Settings > About Phone > tap Build Number repeatedly until the message 'you are now a developer' appears) and USB debugging from your phone (Settings > Developer Options > USB debugging).

For iOS, you need a mac and install Xcode (available from the app store). This tutorial assumes an Android device, although almost all steps are the same.

Step 2: Creating a Project in Unity

• Open up Unity, and create a new project
• Pick a name for your project
• Select a location to save it to (a project is basically a folder)
• Select the 3D option, rather than 2D.
• It is not necessary to import assets now, and generally it is advisable to keep the assets to a minimum as they are all copied into the project directory.
• After clicking 'create project', the main Unity interface will appear. The core components of the interface are explained in the image above.

Step 3: Enabling Cardboard Support

• Right click the Project view > import package > custom package
• Select the Cardboard SDK that you downloaded earlier
• Click import with all assets checked

The empty unity project all-ready comes with two components in the hierarchy: a camera and a directional light (sun). However, Cardboard provides its own camera with a stereoscopic view and optical distortion.

• Delete the main camera from the hierarchy
• In the project view, navigate to Cardboard > Prefabs, and drag the CardboardMain prefab (blue box) to the hierarchy view. A prefab is a pre-made component that already has a defined purpose, and is made to be easily re-used.
• To have something to look at, let's place a simple cube in front of the camera. Right click the hierarchy > 3D object > Cube. Use the move tool (four-way icon in top-left corner) and move the cube in front of the camera in the scene view. Also use the rotate tool to rotate the cube.
• Now press the play button in the top of the window. You will be presented with the VR environment in the game view, rendered from the two view points (it should look something like the screenshot above). By holding alt while moving the cursor you can look around.

Now save the scene, by clicking file > save scene. It is advisable to create a scenes folder for the sake of keeping things neat.

Step 4: Loading the Scene on Your Phone

That was quite straight-forward. Now let's put it on a device so it can actually be tested in VR.

• If, besides the CardboardMain object, there is also a component called just 'Cardboard' in the hierarchy, remove it (it will cause a freeze upon starting the app).
• First, select the platform you want the app to run on by going to File > Build settings and clicking 'Switch platform' with the desired platform selected.
• Now click 'player settings'. The specifics for building the app for the selected platform will appear in the inspector.
• Fill in a unique identifier for your app in Other Settings > Bundle identifier, using the reverse domain name notation (e.g. com.[yourName].[nameOfYourApp], without spaces).
• Make sure Unity knows the location of your Android SDK by going to Preferences > External Tools and checking whether 'Android SDK Location' is set correctly.
• Connect your phone to your computer with a USB cable.
• Click File > Build & Run. Save the build in the project folder (or wherever you desire).

If everything went as it should, you should now have the scene running on your phone. Try it out by placing it in your Cardboard headset.

Step 5: Moving Around

You will quickly feel constrained, standing in just a single position in VR. There are some next-level techniques coming up in tracking your physical movement and using that in the VR environment (see, for instance, the HTC Vive). For the sake of portability and simplicity, for now the magnetic Cardboard trigger will be used to walk in the direction you are looking. First, we need to give the player some weight and size.This will ensure the environment is perceived as spacious as it would be in reality, that you cannot walk through walls and stay on the ground. Since we will not see this character (except perhaps its shadow), a simple capsule shape will suffice.

• Add the character by clicking CameObject > 3D Object > Capsule, and name it something like 'player'.
• In the inspector, set 'y' to 0.9 and 'height' to 1.8 (these units are in meters)
• Click 'add component' in the inspector, and type 'Rigidbody' (select the non-2D option). In the components 'constraints' option, check 'Freeze Rotation' for x, y and z (to ensure the character doesn't fall over).
• Drag the CardboardMain object on top of the player capsule (making it a 'child' of the capsule). Use the move tool to place the camera approximately at eye-height.
• If you press play now, you will notice that you fall down immediately. This is because the player is now subject to gravity, and needs a floor to support it. A simple way to do this is by creating a plane (GameObject > 3D Object > Plane), placing it at position 0,0,0 and scaling it to 10,10,10.

To move around using the Cardboard trigger, a script strongly based on a solution from ITP-VR will be used. The script can be downloaded below.

• Create a Scripts folder in the project / assets view, and drag the script in there.
• Then take the newly imported PlayerControl script, and drag it on top of the Player object in the hierarchy.
• With the player object selected, click the circle next to the 'Cam' property of the 'Player Control' component in the inspector. Select the 'Main Camera' as camera. The script uses this camera to determine the direction to walk in.
• Set 'Current Target Speed' to the desired speed. 2 or 3 seems to yield decent results. Since speed is applied through a force, a higher speed will also result in an exponentially longer distance traveled per click.

Now you can walk around by using the WASD keys. To use the trigger of the Cardboard headset, and make development easier later on, we should import Cardboard Controls+, a Unity package that adds easy interfacing with the controls available.

• Download the latest Unity package from https://github.com/JScott/cardboard-controls/releases/latest
• Go to Project > Import Package > Custom Package and import everything except the demo folder.
• Go to CardboardControl > Prefabs and drag 'CardboardControlManager' to the hierarchy.

Step 6: Creating the Room

In 3D applications and games, often there is a trade-off between graphics and performance. This is especially important on mobile, given the limited computational power. Still, there is a wide set of tricks available to make something performant and beautiful. General tips include:

1. Keeping the polygon / vertex count of the in-game objects low.
2. Pre-calculate the lighting and reflections of whatever is not dynamic (baking).
3. Use simplified real-time lighting calculations (light probes and reflection probes)
4. Limit the amount of rendered objects through fog and occlusion culling
5. Keep textures small if they are not seen from up-close

In general, for a pleasant VR experience, a constantly high frame-rate (50+ fps) is more important than pretty pictures. Lagging and jittering also may cause dizziness for those wearing the headset.

In this step I encourage you to be creative, and design some environment that you have in mind (model in your favourite CAD or modelling suite, convert your 3D models to one of the accepted formats, and check whether your polygon count can be lowered using software like Meshlab). As this tutorial is mostly proof of concept, I will make a simple example: an interactive version of the stairs room in Drake's Hotline Bling video (minus the silly dancing).

• I created a prefab for a simple 20 x 20 x 5 room, made from one-sided planes (easy to look inside). Download room.prefab, create a 'prefabs' folder in your assets, import it and place it in the scene (you can delete the existing plane and cube).
• Download stairs.obj, simple stairs I modelled, and place it in the scene. Make sure it touches the ground and is placed near the back of the room. Select the object_1 component of the stairs in the hierarchy, click 'Add Component' and type Mesh Collider. This uses the shape of the stairs as a collider, so the player can walk on them.
• Add a plane on top of the stairs, set the scale to (x: 0.4, y: 1, z: 0.3)) and set z-rotation to 180.
• Make the plan a child of the stairs object and, with the stairs object selected, check 'Static' in the top right of the Inspector. This tells Unity that the object is not going to move, and makes calculation more efficient. If asked to change children, click 'yes'.
• In the project view, create a folder named 'materials'. Open it, right click > Create > Material.
• Drag the material on the plane you created on the top of the stairs. With the material selected, adjust the emission setting in the inspector panel. I set it to a pinkish hue, with an intensity around 1.6.
• A directional light source doesn't make sense in a closed-off room. Remove it, and set 'Reflection Intensity' and 'Ambient Intensity' in Window > Lighting > Scene > Environment Lighting to 0. The room should now be solely lit by the 'ceiling gap'.

Step 7: Lighting and Shading the Environment

Let's try to simulate the look of the video while keeping performance high.

• There are no really dark places in the room, and it has a blue / purple hue. This can be simulated using the a set of point lights (GameObject > Light > Point Light). Give them the desired color, and set shadow type to 'Soft Shadow'.
• A second light panel should be added around the ceiling hole, with lesser intensity. Just copy the existing one, and adjust the scale. Make a new material with less intense emission in a more neutral color, and apply it.
• Create a material and apply it to all walls. Give some smoothness to the material, and set the albedo color to a blueish white
• Improve the quality of shadows by going to Edit > Project Settings > Quality and settings Shadow distance to 20. This means that Unity will not bother calculating shadows further away. If your device can handle it, you may also set shadow resolution to something higher.
• Add some fog by going to Window > Lighting > Scene and checking Fog (density around 0.05 and purplish white as colour).

An important part of getting high quality, real-time lighting without losing performance are light probes and reflection probes. These are basically invisible objects in the scene, that sample the lighting and apply it to the object around them.

• Start a light probe group by clicking GameObject > Light > Light Probe Group. Set position to 0,0,0.
• Click 'select all' in the inspector panel
• Click 'duplicate selected'
• Use the move tool to move the selected probes over (see animation)
• Repeat, until the space is filled up (you can also scale up the entire group).
• Add a reflection probe by clicking GameObject > Light > Reflection Probe
• Place a few probes around the scene (4 should be fine), and the size.

Step 8: Interaction and Sounds

The Cardboard Controls+ package, mentioned a few steps ago, provides an easy way of interfacing with the headset, and is a good opportunity to dive into the code and create some interactivity.

We will create a black sphere that, when you look at it, changes size, starts emitting particles and changes the color of the light coming from the back wall. For this step we will also use two sounds, one sampled from a free low rumbling sound (the fades were removed), and a sample loop from the instrumental version of Hotline Bling.

Adding the sphere

• Create a new material and apply it to the back wall ('Plane(4)' in the Room prefab), and determine what emission strength you find works well (take any bright color). This will not matter much later, as the value will be set programmatically. Then set the emission intensity to a grey with 0.70 intensity.
• Rename Plane(4) to 'backwall' (this exact name will be used in the code).
• Place a sphere (GameObject > 3D Object > Sphere) in the middle of the back wall at scale 2,2,2.
• Give it a new material with 0 smoothness and a black Albedo.

Adding sounds

• Create a 'Sounds' folder in the Library view
• Import the downloaded / selected sounds in this folder (hotline.ogg is attached below)
• Select the ceiling hole of the stairs
• In the inspector, click 'add component' and type Audio Source.
• Set the AudioClip property to the imported sound
• Check 'Loop' and 'Play On Awake'
• Set 'Spatial Blend' to 1, to make it a sound with 3-dimensional source, the ceiling hole.
• Set 'Max Distance' somewhere around 13.
• Set the volume curve as illustrated.
• Now select the black sphere, and apply an audio source with the rumble noise as clip.
• Uncheck 'Loop' and 'Play On Awake' (we don't want continuous or random rumbles, it will be triggered in the code)
• Leave the Spatial Blend at 0.

Adding particle system

• Create a new material
• In the inspector, change the Shader property (on the top) from Standard to Particles > Additive (soft). This is an optical effect that brightens elements below it only if it is brighter than those elements (similar to Photoshop blending modes).
• Select the black sphere and add a 'Particle System' component.
• Rotate the sphere a such that the emitter is oriented towards the room.
• In the Particle System component set Shape > Angle to 41, and in the Renderer section set
  • Render Mode: Mesh
  • Mesh: Sphere
  • Material: [select your particle material]
  • Cast Shadows: Off
  • Receive Shadows: off

Code

Now it is time to add some new code. Select the 'Player' object and double-click the 'PlayerControl' script. It will open in MonoDevelop, a code editor for Unity that is integrated. For copying code, I recommend to copy from the final file on Github, because Instructables has some trouble with the formatting.

Add the following before the start function ('void Start(){ ... }'), after 'private CapsuleCollider playerCollider;'

private GameObject backWall;
private GameObject sphere;
private AudioSource sphereSound;
private Color desiredColor;

Append the following inside the start function:

cardboard.gaze.OnChange += CardboardFocusChanges; // Execute when focus changes
backWall = GameObject.Find ("backwall"); // define the back wall object
sphere = GameObject.Find ("Sphere"); // define the black sphere
sphereSound = sphere.GetComponent (); // define the sound player of the sphere
desiredColor = new Color (0f, 0f, 0f); // define the starting color of the backWall (black)

The first line of the previous block call the 'CardboardFocusChanges' function, but that is not defined yet. Let's first get on that. Place the following below the Start function (so after its closing '}', before the FixedUpdate function)

private void CardboardFocusChanges(object sender){
        CardboardControlGaze gaze = sender as CardboardControlGaze; //Cardboard camera gaze
        if (gaze.IsHeld() && gaze.Object() == sphere) {
            float redOrBlue = Random.value; //random value for color
            desiredColor= new Color(redOrBlue, Random.value, 1 - redOrBlue); // set desired color
	}
}

So now, four things need to happen:

1. Animating the color and emission of the back wall to the desired color
2. Increasing the size of the sphere when it is looked at
3. Activating / deactivating sounds
4. Activating / deactivating the particle system

As these things need to happen dynamically, the first three animated even, they are put in a default function that is called every frame: the update function. In this case it is as follows:

void Update(){
        if (backWall.GetComponent<Renderer>().material.color != desiredColor) {            
backWall.GetComponent<Renderer>().material.color = Color.Lerp(backWall.GetComponent<Renderer> ().material.color, desiredColor, Time.deltaTime);  // Animate wall color          
DynamicGI.SetEmissive (backWall.GetComponent<renderer>(), backWall.GetComponent ().material.color * 1.5f); // Animate wall emission
}
if (cardboard.gaze.IsHeld () && cardboard.gaze.Object () == sphere) {
sphere.transform.localScale = Vector3.Lerp (sphere.transform.localScale, new Vector3 (4, 4, 4), Time.deltaTime); // animate sphere scale
sphere.GetComponent<ParticleSystem>().startColor = desiredColor; // set particle color
sphere.GetComponent<ParticleSystem>().enableEmission = true; // enable emission
sphereSound.volume = Mathf.Lerp (sphereSound.volume, 1f, Time.deltaTime); // fade sound on
if(!sphereSound.isPlaying){
sphereSound.time = 0; // reset playback of sound
sphereSound.Play(); // play sound
}
} else {
            sphere.transform.localScale = Vector3.Lerp (sphere.transform.localScale, new Vector3(2,2,2), Time.deltaTime); // animate scale back upon lost gaze
            sphere.GetComponent<ParticleSystem>().enableEmission = false; // disable particles
            sphereSound.volume = Mathf.Lerp (sphereSound.volume, 0f, Time.deltaTime); // fadeout
}
}

Step 9: Final Result

Source code: https://github.com/timmevandermeer/CardboardTutorialHotline

Try it out on your device (installable .apk):

https://github.com/timmevandermeer/CardboardTutori...

VR as a technique for prototyping real world interactive environments
I do think, still, that there is great potential in VR for creating immersive experiences with a low investment. In this case the shortcomings lie in other experiential levels, such as physical movement, smell and touch. The next generation of VR devices tackle these issues, but often still at the expense of portability.

Unity and Cardboard have proven versatile tools, and personally I think they are still packaged in a user-friendly (i.e. developer-friendly) way. On the other hand, it is a bit more hardcore than the 3D modelling and 2D prototyping tools that most designers try their hands on. With some persistence, Unity can be learned. Also, it allows for plenty of ways to think visually and be creative, rather than diving deep into code.

Google Cardboard is a fun toy to play with, but does lack behind the more premium VR headsets in terms of immersion, including the Samsung Gear VR. It does however do what prototyping is supposed to do: communicate something vividly and quickly, without high investment.