Boost Your Memory With an Augmented Reality Mind Palace

The use of mind palaces, like in Sherlock Holmes, has been used by memory champions to recall a lot of information such as the order of the cards in a shuffled deck. The mind palace or method of loci is a memory technique where visual mnemonics are pinned to physical locations and are visualized in specific order to recall information.

A common technique used by memory champions is standing on the doorway of a room in one's house and choosing 5 distinct large items and ordering them from 1 to 5. Then, repeating the process for another room with items from 6 to 10. Once one is familiar with these items, then visual mnemonics are chosen and placed on this objects in the order they are to be memorized. The more unique these visual mnemonics are, the easier they are to remember.

I am inspired by the advancements in augmented reality which gives us the opportunity to 'touch' the untouchable. Rather than seeing with the mind's eye, it allows us to see it in our physical world, as well as interact and 'touch' with augmented objects with our hands. With the concept of a mind palace, whereas rather than visualizing tens or hundreds of objects, they can be realistically projected onto real space.

This is currently in progress a video / gif will be added soon.

Step 1: Software and Hardware Requirements

The following software packages are necessary for running an Augmented Reality App

Hardware

* Android (you may use other OS, but this guide will be based on Android)

* Windows

System Requirements for Unity version 5.6.1

OS: Windows 7 SP1+, 8, 10; Mac OS X 10.8+.

GPU: Graphics card with DX9 (shader model 3.0) or DX11 with feature level 9.3 capabilities.

Step 2: Creating a 3D Model

1. Take pictures of the desired training model 360 degrees at around 15 degree increments around the model and a picture from the top. Do not move the model when taking pictures. You also have a maximum of 50 pictures in the trial version of Autodesk Recap360 and it is recommended to take at least 20.

2. Upload all images to Autodesk Recap360 to create a 3D model.

3. Clean up the 3D model using Autodesk Remake

Note this step takes quite a bit of time for Autodesk to reconstruct a 3D model so you can proceed to the next step.

Step 3: Choosing the Target Objects

Target Objects are the physical objects upon which you will pin the augmented reality onto.

In a mind palace, the only requirement are that these objects be large and distinct such that they are easier to remember and visualize. However, with the limitation of augmented reality tools available today, choose objects with high feature textures and distinct edges to make it easier to track and identify.

Step 4: Create Target Images Database on Vuforia

1. Select Develop >> Target Manager

2. Select "Add Database"; Name the database "target_images"; Select the type "Device"

3. Click on the Database you just created

4. Select "Add Target"; select type "3D Object"; Select the 3D object you reconstructed in the previous step; and name the file appropriately

You may have also noticed that there are alternatives to using a 3D object. One of the most tracking-accurate methods is simply to use a 2D paper image, but this will not be a helpful marker to remember for the mind palace, so it is best to use a 3D object that you are already familiar with in your house.

Step 5: Familiarizing With the Unity Environment

* The red box or hierarchy tab is where all the game object, target objects, and main camera will be displayed.

* The blue box is where the project and console tabs are located. The project tab is where you can view your assets. These are the files you are or can use in a game or application. The console tab outputs the warnings, error, and debugging messages.

* The inspector tab on the orange box is where you can find details of particular game objects, target objects, and the camera. It is where you can manipulate individual elements of the game such as transforming the size, rotating, or moving and object numerically, add scripts, change colour, etc.

Step 6: Create a New Unity Project

1. Create a new project; name it appropriately; and select "3D"

2. Import target images database folder as a custom package in Unity under Assets>>Package>>Custom Package...

3. Add the AR Camera and Image Target

4. Add a model of the object you want to augments. You may CAD it yourself or download one from the Asset Store if available.

5. Drag this game object model under the image target in the hierarchy.

6. Add the database Load Behaviour Script onto the Camera and check activate

Step 7: Running the Demo on an Android

* Make sure that you have the Android Unity Plugin installed

1. Go to Inspector >> Player Setting and change the Bundle Identifier to be com.YOURCOMPANYNAME.YOURAPPNAME

2. Go to File >> Build Settings and click "add open scenes"

3. Click Build and Run; name it appropriate; save the .apk file.

Unity will launch as an android application

Step 8: Adding MultiScene Capability

What we have done in the previous steps is a single scene and single object. However, this will not help us memorize a lot of information, specifically only one, so we need to create a script that allows for multiple scenes.

1. Create a new game object and name it "scene changer."

2. Download the C# file, I've coded and import it as an asset (or drag and drop into the assets/scripts folder).

3. Attach this script onto the game object "scene changer" by dragging it to the inspector tab of "scene changer."

4. Duplicate current scene that you have and rename the other scene appropriately. Do not name it with spaces.

5. Specify the next scene in the Inspector tab from the name of the scene.

6. Add a new database (based on a different training set) Load Behaviour Script onto the Camera and check activate

7. Drag a new game object model under the image target which corresponds to the training set from the database.