Introduction: AR Portal to the Upside Down From Stranger Things
This Instructable will go through creating an augmented reality mobile app for the iPhone with a portal that leads to the upside down from the Stranger Things. You can go inside the portal, walk around, and come back out. Everything inside the portal can only be seen through the portal until you walk inside. Once inside, everything will render everywhere, until you walk back through into the real world. We will use the Unity 3D video game engine with the Apple ARKit plugin. All the software we will use can be downloaded and used for free. You don't need to be an expert to follow along, we will go through every step!
Step 1: Start a New Unity Project.
First off, download Unity3D and make sure to install the build files for the IOS platform. You will also need to download Xcode and sign up for a free apple developer account. Your iPhone will also have to be running IOS 11 or greater. As of today Febuary 5th 2018, IOS 11.3 is out but xCode 9.2 does not yet have support files for it. So if you are running the very latest IOS version make sure to download the latest Xcode beta version from Apple.Developer.com.
Once you have all the necessary programs, open up Unity and start a new project, call it whatever you want. We are going to need the Apple ARKit plugin so that we can use our phone's camera to detect the ground an place objects on the floor. Let's import that now by going to the Asset Store tab and search "ARKit". You will need to create a free Unity account if you don't already have one, then click import to get the plugin.
Navigate to the examples folder in the ARKit folder and find the "UnityARKitScene." Double click that to open it. We are going to use this scene as a starting point and build out from here. This scene by default will allow you to detect the ground and when you tap the screen, a cube will be placed in that position.
Lets first get our build settings squared away so we don't forget to do it later. Click file, build settings and remove all scenes from that list. Click add open scenes to add our current one. The last thing we need to set up here is in player settings go down to bundle identifier and the format for this string is com.YourCompanyName.YourAppName, so in my case I do something like com.MatthewHallberg.PortalTest.
Step 2: Set Up the Scene.
First take a look to the left and find the game object called "GeneratePlanes". With that highlighted, look off to the right now and click the check box to disable it. This way we don't have the ugly blue squares being generated when ARKit detects a ground plane. Next delete the "RandomCube" game object because we don't want to see that in our scene.
Now we need to first create our portal doorway. Delete the cube that is a child of the "HitCubeParent". Right click and choose create empty game object. Rename it "Portal". Now right click on that object and create a cube, this will make it a child of the portal. Rename it "PostLeft" and this will be the left post of our portal. Scale it so the x is 1 the y is 28 and the z is one. Do the same thing for the right post. Now create the top post and scale the y to 14. Turn this sideways and move it such that it connects the other posts. Make the entire portal scale 1.3 x 1.4 x 1.
Go to google and type in wood or bark texture. Download one of those images and drag it into your assets folder in Unity. Now drag that image onto all of your portal posts.
Click on the "Portal" object again and click add component on the right. Add the "UnityARHitTestExample" script to it. There is an empty slot there for "Hit Transform", drag the "HitCubeParent" object into that slot.
Step 3: Let's Make Some Particles.
Now we are going to use the Unity Particle system to make a smoke and floating particle effect for inside our portal. Go to Assets at the top menu bar, standard assets, and import particle systems.
Create two empty game objects inside your portal and call one "SmokeParticles" and the other one "FloatingParticles."
Add a particle system component to the smoke particles.
This component has a bunch of options but we only need to change a couple.
Change the start color to something dark blue with about 50% transparency. Make the emission rate 100. Inside shape, make the radius .01. In the renderer portion at the bottom change min size to .8 and max size to 5. On the material component just choose the smoke material from the list, but we are going to change this later.
Add a particle system to the floating particles game object now and set the emission to 500. Set start lifetime to 2, radius to 10, min particle size to .01, and max particle size to .015. Set the material to default particle for now.
Finally take both game objects and rotate them by 90 degrees on the x and raise them up into the air so they are emitting down onto the portal doorway.
Step 4: Slowing Down the Particles.
Since we want these particles to cover a large area but also move slow we need to create our own sample function. So right click in the assets folder and create a new C# script and call it "ParticleSample." Copy and paste in this code:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class ParticleSample : MonoBehaviour {
private ParticleSystem ps;
// Use this for initialization
void Start () {
ps = GetComponent ();
StartCoroutine (SampleParticleRoutine ());
}
IEnumerator SampleParticleRoutine(){
var main = ps.main;
main.simulationSpeed = 1000f;
ps.Play ();
yield return new WaitForSeconds (.1f);
main.simulationSpeed = .05f;
}
}
Now drag this script onto each of your particle system game objects.
Step 5: Creating the Portal!
Now we need to create the portal so right click on the portal game object and create a quad. Scale the quad so it covers the entire portal, this is going to become our portal window. First thing we need to add to it is the portal shader, this will only render objects with another specific shader on them. Right click in the assets folder and create a new unlit shader. Remove everything in there and paste in this code:
Shader "Portal/portalWindow"
{
SubShader
{
Zwrite off
Colormask 0
cull off
Stencil{
Ref 1
Pass replace
}
Pass
{
}
}
}
Right click in the hierarchy and create a new material, call it PortalWindowMat, in the dropdown for this material find the portal section, and choose portal window. Drag this material onto your portal quad.
Step 6: Particle Shaders.
Right click in the assets folder again and create a new shader. We need to make the shaders for the particles that go inside the portal. Replace all the code with this:
Shader "Portal/Particles" {
Properties {
_TintColor ("Tint Color", Color) = (0.5,0.5,0.5,0.5)
_MainTex ("Particle Texture", 2D) = "white" {}
_InvFade ("Soft Particles Factor", Range(0.01,3.0)) = 1.0
_Stencil("stencil", int) = 6
}
Category {
Tags { "Queue"="Transparent" "IgnoreProjector"="True" "RenderType"="Transparent" "PreviewType"="Plane" }
Blend SrcAlpha OneMinusSrcAlpha
ColorMask RGB
Cull Off Lighting Off ZWrite Off
SubShader {
Stencil{
Ref 1
Comp[_Stencil]
}
Pass {
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#pragma target 2.0
#pragma multi_compile_particles
#pragma multi_compile_fog
#include "UnityCG.cginc"
sampler2D _MainTex;
fixed4 _TintColor;
struct appdata_t {
float4 vertex : POSITION;
fixed4 color : COLOR;
float2 texcoord : TEXCOORD0;
UNITY_VERTEX_INPUT_INSTANCE_ID
};
struct v2f {
float4 vertex : SV_POSITION;
fixed4 color : COLOR;
float2 texcoord : TEXCOORD0;
UNITY_FOG_COORDS(1)
#ifdef SOFTPARTICLES_ON
float4 projPos : TEXCOORD2;
#endif
UNITY_VERTEX_OUTPUT_STEREO
};
float4 _MainTex_ST;
v2f vert (appdata_t v)
{
v2f o;
UNITY_SETUP_INSTANCE_ID(v);
UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
o.vertex = UnityObjectToClipPos(v.vertex);
#ifdef SOFTPARTICLES_ON
o.projPos = ComputeScreenPos (o.vertex);
COMPUTE_EYEDEPTH(o.projPos.z);
#endif
o.color = v.color * _TintColor;
o.texcoord = TRANSFORM_TEX(v.texcoord,_MainTex);
UNITY_TRANSFER_FOG(o,o.vertex);
return o;
}
UNITY_DECLARE_DEPTH_TEXTURE(_CameraDepthTexture);
float _InvFade;
fixed4 frag (v2f i) : SV_Target
{
#ifdef SOFTPARTICLES_ON
float sceneZ = LinearEyeDepth (SAMPLE_DEPTH_TEXTURE_PROJ(_CameraDepthTexture, UNITY_PROJ_COORD(i.projPos)));
float partZ = i.projPos.z;
float fade = saturate (_InvFade * (sceneZ-partZ));
i.color.a *= fade;
#endif
fixed4 col = 2.0f * i.color * tex2D(_MainTex, i.texcoord);
UNITY_APPLY_FOG(i.fogCoord, col);
return col;
}
ENDCG
}
}
}
}
Create two new materials, one called portalSmoke, and one called portalParticles.
For each one choose this shader, from the drop down, in portals, particles. For the smoke particles choose a smoke texture and for the particles choose the particle texture. Change the color of the smoke to a darker blue with about 50% transparency. Go to the renderer component of each particle system in your portal and choose their respective materials that we just created.
Step 7: Create the Skybox.
Now to really create the upside down type of look we have to tint everything dark blue. For this we will use a transparent skybox so make a new shader and paste in this code:
Shader "Portal/portalSkybox" {
Properties {
_Tint ("Tint Color", Color) = (.5, .5, .5, .5)
[Gamma] _Exposure ("Exposure", Range(0, 8)) = 1.0
_Rotation ("Rotation", Range(0, 360)) = 0
[NoScaleOffset] _Tex ("Cubemap (HDR)", Cube) = "grey" {}
_Stencil("StencilNum", int) = 6
}
SubShader {
Tags { "Queue"="Background" "RenderType"="Background" "PreviewType"="Skybox" }
Cull Off ZWrite Off
Blend SrcAlpha OneMinusSrcAlpha
Stencil{
Ref 1
Comp[_Stencil]
}
Pass {
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#pragma target 2.0
#include "UnityCG.cginc"
samplerCUBE _Tex;
half4 _Tex_HDR;
half4 _Tint;
half _Exposure;
float _Rotation;
float3 RotateAroundYInDegrees (float3 vertex, float degrees)
{
float alpha = degrees * UNITY_PI / 180.0;
float sina, cosa;
sincos(alpha, sina, cosa);
float2x2 m = float2x2(cosa, -sina, sina, cosa);
return float3(mul(m, vertex.xz), vertex.y).xzy;
}
struct appdata_t {
float4 vertex : POSITION;
UNITY_VERTEX_INPUT_INSTANCE_ID
};
struct v2f {
float4 vertex : SV_POSITION;
float3 texcoord : TEXCOORD0;
UNITY_VERTEX_OUTPUT_STEREO
};
v2f vert (appdata_t v)
{
v2f o;
UNITY_SETUP_INSTANCE_ID(v);
UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(o);
float3 rotated = RotateAroundYInDegrees(v.vertex, _Rotation);
o.vertex = UnityObjectToClipPos(rotated);
o.texcoord = v.vertex.xyz;
return o;
}
fixed4 frag (v2f i) : SV_Target
{
half4 tex = texCUBE (_Tex, i.texcoord);
half3 c = DecodeHDR (tex, _Tex_HDR);
c = c * _Tint.rgb * unity_ColorSpaceDouble.rgb;
c *= _Exposure;
return half4(c, .5);
}
ENDCG
}
}
Fallback Off
}
Now create a new skybox material, call it "PortalSkybox" and choose this portalSkybox shader from the portal menu. Go to Window, Lighting, at the top and choose this skybox we just created. Go to the main camera and set clear flags to skybox. While we are here lets add some components on our camera so we can detect collisions. Add a rigidbody component to the camera and uncheck use gravity. Add a box collider and check on is trigger. Make the box colliders size .5 x 1 x 4. Set the clipping plane on the camera to .01.
Step 8: Portal Logic.
The last thing we need to do is create the logic that controls our portal. Create a new C# script and call it PortalController.
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
namespace UnityEngine.XR.iOS{
public class PortalController : MonoBehaviour {
public Material[] materials;
public MeshRenderer meshRenderer;
public UnityARVideo UnityARVideo;
private bool isInside = false;
private bool isOutside = true;
// Use this for initialization
void Start () {
OutsidePortal ();
}
void OnTriggerStay(Collider col){
Vector3 playerPos = Camera.main.transform.position +
Camera.main.transform.forward * (Camera.main.nearClipPlane * 4);
if (transform.InverseTransformPoint(playerPos).z <= 0){
if (isOutside) {
isOutside = false;
isInside = true;
InsidePortal ();
}
} else {
if (isInside) {
isInside = false;
isOutside = true;
OutsidePortal ();
}
}
}
void OutsidePortal(){
StartCoroutine (DelayChangeMat (3));
}
void InsidePortal(){
StartCoroutine (DelayChangeMat (6));
}
IEnumerator DelayChangeMat(int stencilNum){
UnityARVideo.shouldRender = false;
yield return new WaitForEndOfFrame ();
meshRenderer.enabled = false;
foreach (Material mat in materials) {
mat.SetInt ("_Stencil", stencilNum);
}
yield return new WaitForEndOfFrame ();
meshRenderer.enabled = true;
UnityARVideo.shouldRender = true;
}
}
}
Drag this new script onto your portal window. This will transition us in and out of the portal whenever the collider on our camera is colliding with the portal window. Now in the function that changes all the materials we tell the ARkit plugin to not render the frame, so go to the main camera and open the UnityARVideo script. Create a public bool shouldRender at the top and set it equal to true. Down in the OnPreRender() function wrap everything in an if statement where everything inside will only run if shouldRender is true. The whole script should look like this:
using System;
using System.Runtime.InteropServices;
using UnityEngine;
using UnityEngine.Rendering;
namespace UnityEngine.XR.iOS
{
public class UnityARVideo : MonoBehaviour
{
public Material m_ClearMaterial;
[HideInInspector]
public bool shouldRender = true;
private CommandBuffer m_VideoCommandBuffer;
private Texture2D _videoTextureY;
private Texture2D _videoTextureCbCr;
private Matrix4x4 _displayTransform;
private bool bCommandBufferInitialized;
public void Start()
{
UnityARSessionNativeInterface.ARFrameUpdatedEvent += UpdateFrame;
bCommandBufferInitialized = false;
}
void UpdateFrame(UnityARCamera cam)
{
_displayTransform = new Matrix4x4();
_displayTransform.SetColumn(0, cam.displayTransform.column0);
_displayTransform.SetColumn(1, cam.displayTransform.column1);
_displayTransform.SetColumn(2, cam.displayTransform.column2);
_displayTransform.SetColumn(3, cam.displayTransform.column3);
}
void InitializeCommandBuffer()
{
m_VideoCommandBuffer = new CommandBuffer();
m_VideoCommandBuffer.Blit(null, BuiltinRenderTextureType.CurrentActive, m_ClearMaterial);
GetComponent().AddCommandBuffer(CameraEvent.BeforeForwardOpaque, m_VideoCommandBuffer);
bCommandBufferInitialized = true;
}
void OnDestroy()
{
GetComponent().RemoveCommandBuffer(CameraEvent.BeforeForwardOpaque, m_VideoCommandBuffer);
UnityARSessionNativeInterface.ARFrameUpdatedEvent -= UpdateFrame;
bCommandBufferInitialized = false;
}
#if !UNITY_EDITOR
public void OnPreRender()
{
if (shouldRender){
ARTextureHandles handles = UnityARSessionNativeInterface.GetARSessionNativeInterface ().GetARVideoTextureHandles();
if (handles.textureY == System.IntPtr.Zero || handles.textureCbCr == System.IntPtr.Zero)
{
return;
}
if (!bCommandBufferInitialized) {
InitializeCommandBuffer ();
}
Resolution currentResolution = Screen.currentResolution;
// Texture Y
if (_videoTextureY == null) {
_videoTextureY = Texture2D.CreateExternalTexture(currentResolution.width, currentResolution.height,
TextureFormat.R8, false, false, (System.IntPtr)handles.textureY);
_videoTextureY.filterMode = FilterMode.Bilinear;
_videoTextureY.wrapMode = TextureWrapMode.Repeat;
m_ClearMaterial.SetTexture("_textureY", _videoTextureY);
}
// Texture CbCr
if (_videoTextureCbCr == null) {
_videoTextureCbCr = Texture2D.CreateExternalTexture(currentResolution.width, currentResolution.height,
TextureFormat.RG16, false, false, (System.IntPtr)handles.textureCbCr);
_videoTextureCbCr.filterMode = FilterMode.Bilinear;
_videoTextureCbCr.wrapMode = TextureWrapMode.Repeat;
m_ClearMaterial.SetTexture("_textureCbCr", _videoTextureCbCr);
}
_videoTextureY.UpdateExternalTexture(handles.textureY);
_videoTextureCbCr.UpdateExternalTexture(handles.textureCbCr);
m_ClearMaterial.SetMatrix("_DisplayTransform", _displayTransform);
}
}
#else
public void SetYTexure(Texture2D YTex)
{
_videoTextureY = YTex;
}
public void SetUVTexure(Texture2D UVTex)
{
_videoTextureCbCr = UVTex;
}
public void OnPreRender()
{
if (!bCommandBufferInitialized) {
InitializeCommandBuffer ();
}
m_ClearMaterial.SetTexture("_textureY", _videoTextureY);
m_ClearMaterial.SetTexture("_textureCbCr", _videoTextureCbCr);
m_ClearMaterial.SetMatrix("_DisplayTransform", _displayTransform);
}
#endif
}
}
Step 9: Almost Done!
Finally when we click the screen and place the portal we want it to always face us. To do this go to the "UnityARHitTestExample" script on the portal. Replace everything inside with this:
using System;
using System.Collections.Generic;
namespace UnityEngine.XR.iOS
{
public class UnityARHitTestExample : MonoBehaviour
{
public Transform m_HitTransform;
public float maxRayDistance = 30.0f;
public LayerMask collisionLayer = 1 << 10; //ARKitPlane layer
bool HitTestWithResultType (ARPoint point, ARHitTestResultType resultTypes)
{
List hitResults = UnityARSessionNativeInterface.GetARSessionNativeInterface ().HitTest (point, resultTypes);
if (hitResults.Count > 0) {
foreach (var hitResult in hitResults) {
Debug.Log ("Got hit!");
m_HitTransform.position = UnityARMatrixOps.GetPosition (hitResult.worldTransform);
m_HitTransform.rotation = UnityARMatrixOps.GetRotation (hitResult.worldTransform);
Debug.Log (string.Format ("x:{0:0.######} y:{1:0.######} z:{2:0.######}", m_HitTransform.position.x, m_HitTransform.position.y, m_HitTransform.position.z));
Vector3 currAngle = transform.eulerAngles;
transform.LookAt (Camera.main.transform);
transform.eulerAngles = new Vector3 (currAngle.x,transform.eulerAngles.y,currAngle.z);
return true;
}
}
return false;
}
// Update is called once per frame
void Update () {
#if UNITY_EDITOR //we will only use this script on the editor side, though there is nothing that would prevent it from working on device
if (Input.GetMouseButtonDown (0)) {
Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
RaycastHit hit;
//we'll try to hit one of the plane collider gameobjects that were generated by the plugin
//effectively similar to calling HitTest with ARHitTestResultType.ARHitTestResultTypeExistingPlaneUsingExtent
if (Physics.Raycast (ray, out hit, maxRayDistance, collisionLayer)) {
//we're going to get the position from the contact point
m_HitTransform.position = hit.point;
Debug.Log (string.Format ("x:{0:0.######} y:{1:0.######} z:{2:0.######}", m_HitTransform.position.x, m_HitTransform.position.y, m_HitTransform.position.z));
//and the rotation from the transform of the plane collider
m_HitTransform.rotation = hit.transform.rotation;
}
}
#else
if (Input.touchCount > 0 && m_HitTransform != null)
{
var touch = Input.GetTouch(0);
if (touch.phase == TouchPhase.Began || touch.phase == TouchPhase.Moved)
{
var screenPosition = Camera.main.ScreenToViewportPoint(touch.position);
ARPoint point = new ARPoint {
x = screenPosition.x,
y = screenPosition.y
};
// prioritize reults types
ARHitTestResultType[] resultTypes = {
ARHitTestResultType.ARHitTestResultTypeExistingPlaneUsingExtent,
// if you want to use infinite planes use this:
//ARHitTestResultType.ARHitTestResultTypeExistingPlane,
ARHitTestResultType.ARHitTestResultTypeHorizontalPlane,
ARHitTestResultType.ARHitTestResultTypeFeaturePoint
};
foreach (ARHitTestResultType resultType in resultTypes)
{
if (HitTestWithResultType (point, resultType))
{
return;
}
}
}
}
#endif
}
}
}
Step 10: Put the App on Your Phone!
Finally we are done. Go to file, build settings and click build. Open up Xcode and choose the folder that was created from the build. Choose your development team and put the app on your phone! You may want to change the colors of the particles and skybox in order to suit your needs. Let me know in the comments if you have any questions and thanks for looking!
Participated in the
Epilog Challenge 9
