Augmenting Augmented Reality

Today’s Augmented Reality (AR) devices enable users to interact almost naturally with their surroundings, e.g., by pinning digital content onto real-world objects. However, current AR display are mostly limited to optical and video see-through technologies. Nevertheless, extending Augmented Reality (AR) beyond screens by accommodating additional modalities (e.g., smell or haptics) or additional visuals (e.g., peripheral light) has recently become a trend in HCI. In this tutorial, we provide beginner level, hands-on instructions for augmenting an Augmented Reality application using peripheral hardware to generate multi-sensual stimuli.

Step 1: Motivation: What We Want to Do!

AR apps are often limited to visual representation of the virtual content. In this tutorial, we will augment the typical AR experience with multi-sensual Feedback.

As an application scenario, we choose an AR App to explore the four seasons with multi-sensual feedback. For example feeling cold near a snow man.

Step 2: What You Will Need (Bill of Materials)

1) Hardware Components

• A HoloLens or any of your favourite AR Headsets that can be used with Unity3D.

• A NodeMCU DevKit v2 (ESP8266 -12E module), e.g., available from Amazon, or EXP Tech.

• 2 Wire-to-Board Termination Blocks 1x8 Pin, 2.54 mm. We use those for easy prototyping. If you have a NodeMCU with pre-soldered headers, you may also use your favourite breadboard and jumper wires.

• A selection of wearable actuators. You can practically use whatever actuators you like. In this tutorial we use a 5V Cooling Fan, a Vibration Disc Motor as well as a 2V Peltier Element (you might also use a 5V Peltier Element with a MOSFET Transistor for power supply),

2) Software

• Unity3D 2018.1
• Arduino IDE

3) Tools

• Slot head screwdriver 0.4 mm x 2 mm

• Micro USB Cable

4) Optional (for mounting)

• 3mm acrylic glass

• Velcro One-Wrap (soft, double-sided)

• Velcro Wrap (self-adhesive tape)

Step 3: Setup Unity

To develop our 3D content, we use Unity3D (Unity). Unity is a game engine that allows to create content for various platforms (e.g., Hololens, Smartphones, Tablets, etc). The software can be download on the developers homepage (https://unity3d.com). Please follow the steps of the installer software. When installed please start the software and create a new project.

Step 4: Sample Scene

The left figure shows a simple scene in Unity3D. A cube was added to the scene and rotated on the x-, y-, and z-Axis. In the Inspector on the right-hand side, the values of the rotation are shown and can be edited. Other attributes like position and scale can be edited here as well.

To add a cube to the scene click on GameObject -> 3D Object -> Cube.

To add some functionality to the scene a script was added to the cube. The script is called Rotate as shown below in the Assets panel.

The figure on the right-hand side shows the code that lets the cube rotate.

The cube is rotated around the Up- Vector i.e. Vector3(0, 1, 0).

Further, an angle is specified by which the cube is rotated when the method is called. As the method is called in the Update() - Loop the cube is rotated 1 degree each frame.

To add a script to the cube, select the cube, click on Add Component in the Inspector and select New Script. The script can be edited in Visual Studio by double-clicking it in the Asset-Panel.

You can add the code shown below to your script. Copy it to the Update() - Loop.

Finally, to see the cube when the app starts a camera and a light source is needed. If the camera is selected in the scene a preview of the scene is shown.

To start the scene click on the Play-Button. Now you should see a rotating cube.

<p>this.transform.Rotate(Vector3.up, 1.0f);</p>

Step 5: Deploy to Hololens

To develop for Hololens some additional steps are required in Unity.

Prerequisites: Full Deployment Tutorial

Task 1: Configuring a project for Windows Mixed Reality

1. Select File > Build Settings...
2. Select Universal Windows Platform in the Platform list and click Switch Platform
3. Set SDK to Universal 10
4. Set Target device to Any Device to support immersive headsets or switch to HoloLens
5. Set Build Type to D3D
6. Set UWP SDK to Latest installed

Task 2: Configuring for the immersive view instead of 2D view

1. From the Build Settings... window, open Player Settings...
2. Select the Settings for Universal Windows Platform tab
3. Expand the XR Settings group
4. In the XR Settings section, check the Virtual Reality Supported checkbox to add the Virtual Reality Devices list.
5. In the XR Settings group, confirm that Windows Mixed Reality is listed as a supported device. (this may appear as "Windows Holographic" in older versions of Unity)

Task 3: Adding capabilities to the app

The settings are found in Player Settings > Settings for Universal Windows Platform > Publishing Settings > Capabilities.

Check the following settings in the list:

• SpatialPerception
• WebCam
• Microphone
• InternetClient

Task 4: Adjusting the Unity camera settings

1. In the Hierarchy, select the Main Camera
2. In the Inspector panel, set the transform position to 0, 0, 0 so the location of the users head starts at the Unity world origin.
3. Change Clear Flags to Solid Color.
4. Change the Background color to RGBA 0,0,0,0. Black renders as transparent in HoloLens.
5. Change Clipping Planes - Near to the HoloLens recommended 0.85 (meters).

Task 5: Deploying to Hololens

1. Open the Build Settings in Unity (File -> Build Settings)
2. Press on Build and select a folder (maybe creating a dedicated build folder is useful)
3. Wait until the build is complete
4. When the build folder is shown open the Visual Studio Solution
5. Select x86 as the Solution Platform and Device to directly deploy on the Hololens
6. Connect the Hololens to your PC using the USB cable.
7. Hit Play!

This may take some moments but eventually, the scene will be shown on the Hololens. The objects are registered at a certain position. Take a look around.

Maybe add a rotation script to your objects and re-deploy the app to get a better impression of what is possible with the Hololens and Unity.

Step 6: Setup Arduino

We will use the Arduino IDE to program the NodeMCU developer board. Therefore, please download the Arduino IDE from the developers homepage (https://www.arduino.cc/) and install the software by following the steps of the installer.

After installing the Arduino IDE we need to add the NodeMCU board to the IDE.

1. As a first step, go to File > Preferences and click on Additional Board Manager URLs.
2. Here please add the following URL (http://arduino.esp8266.com/versions/2.4.1/package_esp8266com_index.json).
3. Afterwards, click on Tools > Board: "..." > Boards manager.
4. In the boards manager type NodeMCU into the search field.
5. Install the esp8222 entry that shows up (latest version).

As a next step please make sure that the Arduino IDE is configured as shown in the picture

• Board (NodeMCU 1.0 ESP-12E)
• Flash Size (4 MB)
• Upload Speed (115200)
• Programmer (AVRISP mkll)
  

Step 7: Develop for NodeMCU

In this step we want to use the NodeMCU to control a single vibration motor. To start, connect your PC with the NodeMCU board via an USB cable. Then create a new Arduino project by clicking on File > New. Check the connection between your PC and the NodeMCU by uploading the sample code to the board. If this works without problems we successfully connected the developer board with your PC and can start to develop for the NodeMCU.

Every Arduino program contains two predefined functions: setup and loop. Both are essential and required in almost every program. The first function setup is a function that is called once in the beginning. The moment the NodeMCU is connected to power and is done with booting the firmware, the function setup is called. Therefore, it is useful to initialize variables or pins. Tipp: a board reset is also calling the function setup once after it is done booting. The second function loop is called as long as the NodeMCU is powered. If a function call for loop is done the function is called again. Thereby, it is useful for the main code and can react to user input for example.

Additionally, it is important to understand the Serial class offered by Arduino. This one is used for the communication with a PC over USB. Serial is a useful tool to input or output data to the developer board. Mostly it is used for debugging purposes.

In the code listing below we show an example of code for controlling a vibration motor. The code waits on user input and if it gets some, turns on a vibration motor for one second. Here, we first initialize a variable that stores the pin we connect the vibration motor with. Then, in the setup function, we initialize the pin with the pinMode function and set it as an output pin. Afterwards, we start a serial connection via USB and set the Baudrate to 115200. In setup as well as in loop we add a tiny delay that from our experience helps to get rid of unexpected behaviour of the NodeMCU board we use. In the loop function, we check for user input via serial and if that is the case we set the specified pin to high and after one second to low again.

<p>// Pin the vibration motor is connected to<br>const int motorPin = D4;

// function that is called once at startup
void setup()
{
  // set the pin mode for the vibration motor pin
  pinMode(motorPin, OUTPUT);

  // specify the Baudrat the serial is listening to
  Serial.begin(115200);

  delay(10);
}

// function that is called in a loop
void loop()
{
  // checks if serial is available
  if (Serial.available() > 0) 
  {
    // reads the available serial and drops it
    Serial.read();

    // set the vibration motor pin to high
    digitalWrite(motorPin, HIGH);
    // one second delay
    delay(1000);

    // set the vibration motor pin to low
    digitalWrite(motorPin, LOW);
    // one second delay
    delay(1000);
  }
  delay(10);
}</p>

Step 8: Setup the Communication Between Hololens and NodeMCU

For the communication between the Hololens and the NodeMCU developer board we use WiFi. Besides WiFi, there is also the possibility of using Bluetooth for the connection to the Hololens but therefore another board is required. However, in this tutorial we describe the steps to setup the communication via WiFi.

We want the developer board to function as a WiFi access point and the Hololens to connect as a client to the board. To setup the board as a WiFi access point, we import the three header files WiFiClient.h, ESP8266WiFi.h, and ESP8266WebServer.h. Next, we setup two variables containing the SSID for the WiFi network and the password. The server will then be stored in an object of the class ESP8266WebServer. A number to specify the port the server is listening to is required to instantiate the object.

The function handleRequest is used to parse every incoming request. In the implementation given below the function only knows the parameter x. In case the parameter is 1 the vibration motor is turned on. In every other case it is turned off.

Afterwards, we initialize the WiFi server in the setup function. Again, we first initialize the pin and set the serial to the correct Baudrate. Then we configure the access point with WiFi.softAP(ssid, password) and get a IP with WiFi.softAPIP(). The returned IP is then sent over the serial port. Finally, we just need to specify that the server should listen to the root path and that we want to start it.

In the function loop we keep the access point alive with server.handleClient().

<p>#include "WiFiClient.h"<br>#include "ESP8266WiFi.h"
#include "ESP8266WebServer.h"

// Pin the vibration motor is connected to
const int motorPin = D4;

// SSID and password for WiFi access point
const char *ssid = "YOUR_SSID";
const char *password = "YOUR_PASSWORD";

// Webserver and port it is listing to
ESP8266WebServer server(80);
IPAddress myIP;

// function that handles incoming requests
void handleRequest()
{
  // iterate through all parameter
  for (int i = 0; i < server.args(); i++)
  {
    // if parameter is x
    if (server.argName(i).equals("x"))
    {
      // if parameter x equals 1
      if ((server.arg(i)).toInt() == 1)
        // set the vibration motor pin to high
        digitalWrite(motorPin, HIGH);

      // if parameter is x but does not equal 1
      else
        // set the vibration motor pin to low
        digitalWrite(motorPin, LOW);
    }
  }

  // answer the request
  server.send(200, "text/html", "success");
}

// function that is called once at startup
void setup() 
{
  // set the pin mode for the vibration motor pin
  pinMode(motorPin, OUTPUT);

  // specify the Baudrat the serial is listening to
  Serial.begin(115200);

  // configure the ssid and password
  WiFi.softAP(ssid, password);
  // request the IP
  myIP = WiFi.softAPIP();

  // printing the IP into the serial console
  Serial.print("Listening to IP ");
  Serial.println(myIP);

  // define how to handle requests
  server.on("/", handleRequest);
  // start server
  server.begin();

  delay(10);
}

// function that is called in a loop
void loop() 
{
  // keep the access point alive
  server.handleClient();
  delay(10);
}
</p>

Step 9: Use Vuforia Markers for Augmented Reality

In this extra section, we will describe how one could use Vuforia to advance the AR capabilities of our little demo.

We use Vuforia to extend our project with more sophisticated AR capabilities.

Detailed steps can be found here: Vuforia + HoloLens

1. Add AR Vuforia Camera: GameObject > Vuforia > AR Camera. Delete the existing Camera from the scene.
2. Next, configure the Vuforia for the Hololens: Window > Vuforia Configuration: In section Digital Eyewear select Digital Eyewear as the Device Type and HoloLens as the Device Config.
3. Add a target: Add a Vuforia Image Target that can be detected by the HoloLens to add AR content to the scene. The targets can be added like every other GameObject: GameObject > Vuforia > Image
4. Add an Object to the Image Target: Right-click on the ImageTarget in the hierarchy. Select any 3D-Object you like. For example a sphere. The sphere is now attached to the Image Target and only shown if the camera detects the image in the real world.
5. Save the scene and the project.
6. Build the scene for Hololens. Go to File > Build Settings and click build. Like you did before to deploy apps on the HoloLens.
7. After the build has finished the build folder pops up. Open the Visual Studio solution and start the app by hitting Play. (Note: If any errors occur just try to hit play again to restart the app.)
8. Aim on the Image Target with the HoloLens. For example the astronaut like shown in the figure. You should see whatever you attached to the Image Target now in Augmented Reality.

Try to add more compled GameObjects to the Image Targets to see what is possible with Vuforia and the Hololens.

Step 10: Add Actuators for Multi-Sensual Feedback

Now, we will add actuators to our setup. Simple actuators, such as the Cooling Fan, and the Vibration Disc Motor can be connected directly to the NodeMCU: connect red (+) to a digital out pin (here: D4), and blue/black (-) to ground. In the code above, we create a vibration effect (or cooling effect, if you connected the fan) by setting the digital out (here: D4) to HIGH:

digitalWrite(D4, HIGH);

If we would like to turn the disc motor off again, we set it to LOW:

digitalWrite(D4, LOW);

That's it - you now can control the Vibration Disk motor via the HoloLens application.

What is next? Get creative by using any actuator you like. We tried a 5V Cooling Fan, a Vibration Disc Motor, a RGB NeoPixel Strip, as well as a 2V Peltier Element. The Cooling Fan only receives enough power, if the NodeMCU is powered via USB (not battery). If you would like to add actuators that are using up even more power (e.g., a 5V Peltier Element) you will have to add a MOSFET Transistor for extra power supply, as described here.

We recommend to test your effects first separately, and then combine them with the HoloLens application. As a starter, you can try out different effects based on the Arduino IDE's build-in examples, e.g. "Blink" (which is what we did with the Vibration Disk motor).

NeoPixel strips or NeoPixel rings create also nice "lens flare" effects when mounted inside the Hololens. Have a look at a tutorial here.

Once you are satisfied with your effect, integrate it into handleRequest() :

// function that handles incoming requests

void handleRequest()
{
  // iterate through all parameters
  for (int i = 0; i < server.args(); i++)
  {
    // if parameter is x
    if (server.argName(i).equals("x"))
    {
      // if parameter x equals 1
      if ((server.arg(i)).toInt() == 1)       
	// create any effect you like
      else
        // create another effect or do nothing
    }
  }