Placing AR Objects at GPS Coordinates in Augmented Reality

This instructable is going to go over making a mobile app for placing AR objects at GPS coordinates with ARkit and ARCore using Unity3D. I will walk you through setting up a project I made using Mapbox that allows us to tag messages at particular GPS locations and write them to a GameSparks database.

All the software and projects we will use here are free.

Download Unity 3D here if you don'y already have it:

https://unity3d.com/

Step 1: Hurdles...

So this isn't the easiest app to make with the current limitations of mobile GPS and mobile compass.

To be honest the objects show up where they are supposed to maybe about 50% of time. So the results are not great, but they do become more accurate the closer you get to the objects.

Either way, I think a better way to GPS based AR at this point would be using a map like pokemon go does and when you get close to an object the camera opens and you see only that object right in front of you. In the future I would like to do a full tutorial on a more simple implementation like that.

Either way I have used Mapbox before and I know they have a conversion class that has functions for converting GPS coordinates to Unity world coordinates. I thought I could just pass in some GPS coordinates and they would appear where I wanted them. Didn’t work.

The objects show up correctly relative to each other but they were never in the right place. So I did some research and found that my Unity Camera needed to be aligned with true north in order for the objects to show up in the right place.

The problem is the compass in your phone is never totally accurate.

Step 2: Stupid Compass.

The main problem is when your placing objects far from the camera, if your phones idea of true north is off even a few degrees, objects in the distance will appear very far away.

I almost gave up on this venture but a friend told me about this GitHub project:

https://github.com/mapbox/mapbox-ar-unity

This is an attempt to do "world scale" AR from Mapbox where objects are anchored to GPS coordinates so that Augmented Reality experiences can work outside over long distances instead of just inside a small room in your house.

Their project uses AR Position deltas and GPS Position deltas to calculate an angle which is the offset from the AR camera to true north. So basically take one GPS reading when the user starts the app and have them walk in a straight line and take another GPS reading. Then with these two points you can tell which direction that person was walking (assuming their camera is pointing in the direction they are walking). They found that this was more accurate than using the compass on the phone.

The other issue they try to solve is the drift with ARkit. Tracking is ok inside a small room but when your outside walking long distances AR objects end up drifting far from where they are supposed to be.

Step 3: Finally a Solution!

So this Mapbox project offered a way to get a reliable facing direction relative to true north so all that was left was to just place the objects in the world.

I ended up extending this example a little so I could tag a text message at a particular GPS location and then write that information to a GameSparks database. That is what I am going to show you how to use.

The only problem with this project is that you have to get a good alignment before it can be used.

What I mean by alignment is that since Mapbox uses position deltas to calculate heading, when you first open the app and you have to make sure a ground plane is detected and then walk in a straight line for a while until a proper alignment is calculated.

In this sense UX is another hurdle but in the enhancements of the project they give an example of placing cubes down in front of the user to walk through to ensure a good alignment.

So now that you know the limitations of this project let me show you how to use it so guys can play around. You will ultimately need to create a mapbox account and a gamesparks account both of which are free. I used gamesparks just so I could save GPS coordinates somewhere because during testing you will be restarting and rebuilding the app quite often.

Also, this should work with both ARkit and ARcore but I only have an iPhone at the moment so that was all I could test.

Step 4: Lets Get the App Working!

First things first download this project from GitHub: https://github.com/MatthewHallberg/MapboxGPSHeadin...

Save the file to your desktop and open it up in Unity.

Go to file, build settings, and switch your platform to either Android or IOS.

Now go to Mapbox.com and create an account if you don't already have one.

Grab your API key and go back to Unity, click the Mapbox tab at the top, and click configure. Paste in your API key there.

Now create an account on GameSparks.com and click the button on the top right to create a new app.

Call it whatever you want and copy your API key and app secret. Go back to Unity and find the GameSparks tab at the top. Click on configure and paste your credentials in there as well.

Step 5: Configure GameSparks.

Now before the app will work, we need to configure our database.

GameSparks uses noSQL so we need to first create a collection and add the events that our app uses to read, write, and delete from the server.

First create a new collection on the GameSparks website and call it whatever you want.

Now we need to create 3 events.

The first thing the app needs to do is write message object to the database.

A message has a latitude, longitude, and then the text of the message.

So go to configurator and add a new event. Make the short code "SAVE_GEO_MESSAGE".

Put in whatever you want for the name and description.

Add the short code for 3 attributes:

"LAT"
"LON"

"TEXT"

Set each data type to string and set the default value to 0. Make the default aggregation type of each one to "used in script."

Step 6: Add the Final Events...

The app also needs to load all the messages from the database and read them into Unity when the app starts so that we can place an AR messages that are stored in the database.

Create another event and call it "LOAD_MESSAGE"

This event does not need any attributes. Again put whatever you want for the name and description.

Do the same thing for another event and call it "REMOVE_MESSAGES"

This even does not need any attributes either.

Now the final thing we need to do is add some "cloud code" or server side code that gets called when each event is started from our app.

Step 7: Add the Cloud Code.

In GameSparks go to configurator and cloud code.

We need to add a script to each event that we just made.

In the LOAD_MESSAGE event add this code and save it:

var messageData = Spark.runtimeCollection("GeoMessage");

var cursor = messageData.find();

var allMessages = [];

while(cursor.hasNext()) {

var obj = cursor.next();

delete(obj ["_id"]);

allMessages.push(obj);

}

Spark.setScriptData("all_Messages", allMessages); // return all data

In the REMOVE_MESSAGES event add this code:

var messageData = Spark.runtimeCollection("GeoMessage");

messageData.remove({});

Finally in the SAVE_MESSAGES event add this:

var geoMessageList = Spark.runtimeCollection("GeoMessage");

var messageLat = Spark.getData().LAT;

var messageLon = Spark.getData().LON;

var messageText = Spark.getData().TEXT;

var currentMessage = {

"messLat": messageLat,

"messLon": messageLon,

"messText": messageText,

};

geoMessageList.insert(currentMessage);

Step 8: WE ARE DONE!

This project uses the unique ID of your device to authenticate with the GameSparks server so you can now go back to Unity and click play, and you should see in the console "device authenticated..."

Now go to file, build settings, and click build. If you have never made an app for Android or IOS you may need to set up a development account with either Google or Apple.

Once the app is on your phone you need to first make sure ARKit or ARCore detects a ground plane. Now click the log button in the bottom left corner. Walk forward a few steps and you will see "computed alignment" come up in light blue. Now the UI buttons should appear and if any messages are loaded in your database they will get placed in the world.

To tag a new GPS AR message somewhere walk to the location you want the message to be and click the message icon on the top right of the screen. Type in whatever message you want and click the pin button!