How to Use SurfaceMapperGUI for Basic Projection Mapping Projects

In this Instructable I'll be talking about a tool that I created called SurfaceMapperGUI, which is meant to provide a really barebones interface for projection mapping that can be intuitively picked up within a day.

Many other projection mapping tools are available (and I encourage you to check them out), but in my personal research I felt that there weren't any free tools available that were simple enough for someone to open up and intuitively grasp without significant investment of time and/or a strong prior background with code. Hopefully SurfaceMapperGUI will help more people gain some knowledge of the fundamentals behind projection mapping!

Step 1: Introduction to Projection Mapping

What is projection mapping?

Projection mapping is the process of warping or aligning projected images and videos to match the dimensions of physical objects. Essentially it is the use of real-world objects as projection surfaces and using software to correct for distortions.

How does it work?

The Problem

When you take an image and project it onto an arbitrary object you will find that the image simply doesn’t match up with the object. If the object has angles (or is rotated relative to the camera) the image will be awkwardly stretched and distorted. You can physically move the camera to point directly at the surface you want to project on then move it close enough to fill the surface, but that is a very low-tech solution that breaks down as soon as you want to project onto multiple surfaces simultaneously.

The Solution

Using special software, you can “pre-warp” (map) the image you want to project on the computer so that when it physically is projected onto the object it all appears to be undistorted. Attached to this step are some photos of what this looks like for a simple cardboard box that is placed at an angle to the projector.

Necessary parts

In general, to pull off projection mapping you need four things:

1. A projector
2. A computer with data you want to project
3. Software to warp the data onto real-world objects
4. An object or environment that you want your projected data to match.

In Step 7 I'll talk more in-depth about the projector and the computer.
In Step 3 I'll help you install SurfaceMapperGUI (the software) and get it running.
Steps 8-12 will demonstrate using SurfaceMapperGUI onto different types of objects

Step 2: About SurfaceMapperGUI

SurfaceMapperGUI is a program built with the Processing creative coding environment that adds a user-interface (using ControlP5) to the SurfaceMapper library created by ixagon.

Features

Currently SurfaceMapperGUI supports all of the features available through the standard SurfaceMapper (version 0.9) library, as well as some I've added myself. These features include:

1. Create new surfaces (straight-edged quads or curved-edged bezier surfaces).
2. Load and save layouts.
3. Switch back and forth between rendering and calibrating.
4. Alter the resolution of the surfaces.
5. Alter the horizontal and vertical “force” on the edges of bezier surfaces.
6. Anchor points that are moved close to each other will snap together.
7. Assign pictures or videos as textures to any surface.

Future improvements

SurfaceMapperGUI could be a great tool for simple projection mapping experiments and illustrative purposes, but may lack some important features that would make it more suitable for professional projects. Specifically, I think the following features could add a lot to the project:

1. Create ability to create slideshows using static images. However, properly accounting for transitions and timing could be a pretty hefty amount of work.
2. Add some sort of external cue system to trigger transitions between photos and videos.
3. Add ability to play multiple videos back to back.
4. Add ability to remove surfaces.
5. Add ability to re-order the surfaces to control overlapping.

If you'd like to help with the project, feel free to fork the official Github repo and send me some pull requests!

Note about Shaped

While putting the finishing touches on this Instructable I found out that Ixagon had released a commercial projection mapping program based on the SurfaceMapper library, called Shaped, that includes some of the lacking features I listed above. I didn't know he/they were planning on commercializing the code, but hopefully SurfaceMapperGUI isn't violating anything.

If you'd like to purchase Shaped (for the low, low price of .... $1,300 USD) follow this link. I can't really give an informed opinion on it because the demo offered on the website will not run on my computer.

Step 3: Installing the Software

SurfaceMapperGUI is really just a Processing sketch that adds a user interface (using ControlP5) to the SurfaceMapper library. This means that you will need to set up the Processing environment and a few libraries in order to make it work.

If you do not know how to install libraries in Processing, you can learn about the process here.

1. Download and install Processing 1.5.1
   • Newer versions of Processing do not work with the SurfaceMapper library, so make sure to get version 1.5.1.
2. Download and install the SurfaceMapper library (version at time of writing: 0.9)
3. Download and install the GLGraphicslibrary (version at time of writing: 1.0.0)
4. Download and install the GSVideolibrary (version at time writing: 1.0.0)
5. Download and install the ControlP5library (version at time of writing: 1.5.2)
6. Download the SurfaceMapperGUI source code and extract to your Sketchbook location
7. Open SurfaceMapperGUI from Processing’s File > Sketchbook menu, then click the Run button to start the sketch.

Step 4: Calibration Mode Vs. Render Mode

SurfaceMapperGUI operates in two different modes: calibration mode and render mode.

Calibration mode

All of your work will take place in this mode. You can use the various menus and buttons to add and modify new surfaces to the screen. In the next few steps we'll take a closer look at each of the available menus, as well as how to work with the surfaces that are on the screen.

Render mode

Render mode removes everything seen in calibration mode and replaces the surfaces with the images or videos that are assigned to them. Use this mode to actually present your work.

Step 5: Program Options Menu

In the top right corner of calibration mode is the Program Options menu. This menu allows you do general actions regardless of what surface is selected.

Program Options

1. Create a new quad surface - adds a new quad surface (straight edges) to the screen.
2. Create a new bezier surface - adds a new bezier surface (curved edges) to the screen.
3. Load layout - loads surfaces and textures from pre-saved XML files.
4. Save layout - saves all surface and texture information to XML files.
5. Switch to render mode - goes to render mode for you.

Note about layouts

The Load/Save layout buttons rely on reading and writing a few XML files stored in the sketch's "data" folder to save information about surfaces and textures. Originally, the SurfaceMapper library only writes an XML file that contains basic information about all of the surfaces on the screen (their locations and the locations of their vertices). It didn't store any information about textures, which means you have to manually reassign them everything you load a layout. SurfaceMapperGUI will write additional text files that preserve all of the texture associations for you.

Step 6: Surface Options Menus

At the top-left corner of calibration mode sits the surface option menus. A slightly different menu will appear here depending on the type of the surface that is currently selected (quad or bezier).

Quad Options

1. Quad surface name - the name (number) of the selected surface.
2. Resolution - amount of subdivision within the surface (number of grid lines).
3. Source file - image or video that is mapped to the surface.

Bezier Options

1. Quad surface name - the name (number) of the selected surface.
2. Resolution - amount of subdivision within the surface (number of grid lines).
3. Horizontal force - amount of force applied to the left and right edges.
4. Vertical force - amount of force applied to the top and bottom edges.
5. Source file - image or video that is mapped to the surface.

Note on "Source file" list

The list of files found in the Source file drop-down corresponds to all of the images and videos found in the "data/textures" folder of the SurfaceMapperGUI sketch. Just add or remove content to that folder to update the list (before starting SurfaceMapperGUI).

Step 7: Choosing a Computer and a Projector

Before we jump into some examples, let's talk a little bit about the computer and the projector that you'll need for projection mapping.

Computer

Projection mapping is a very graphically intense process, and therefore benefits first and foremost from a good graphics card. However, a good graphics card is only useful if the computer it is connected to is able to feed it a lot of data quickly, especially if that data is being generated in real-time. Therefore, a good processor and plenty of memory is ideal. With this in mind, I would say that the most important components to look at when choosing a computer for projection mapping are:

1. Graphics card
   • At least 1GB of memory
   • Get the most powerful card you can afford.
   • As many output interfaces as you can get. More interfaces = more projectors that you can use at once (be aware, there are several types and shapes of DVI and HDMI).
2. Processor
   • Faster speeds and multiple cores will help your computer do more things at once. This is especially critical if you are generating visuals in real-time, instead of just playing back pre-rendered videos.
3. Memory
   • More memory will allow you to store more image data (and higher quality data) per frame.
   • At least 4GB (8GB or more if you can afford it)

Projector

You can have the most amazing visuals in the world stored on your computer, but if your projector isn’t bright enough, or is out of focus, no one will know! The projector itself is perhaps the most important aspect of projection mapping, which presents somewhat of a “barrier to entry”; projectors are expensive!

1. Brightness (measured in lumens) – how much light the projector puts out. This is one of the most critical parameters for choosing a projector for projection mapping, and can make or break the illusion you are trying to pull off. Cheap conference room and home theater projectors can work great for dark indoor settings, but will not cut it for outdoor performances.
2. Resolution – most projectors have a selection of resolutions you can choose from. Higher resolutions will result in higher-quality images, but will also require higher-quality source data (which can tax your computer). The most important thing is that your source data’s resolution matches the resolution of the projector. Go as high as you think your computer will let you, as lower resolutions produce noticeable pixelation.
3. Throw distance (and image size) – the lens of your projector will dictate the size of the projected image at certain distances, which is usually indicated by a table included in the projector’s user manual. You can also use online tools to find the throw distance (and many other cool parameters) of different projectors, like this one: http://www.projectorcentral.com/projection-calculator.cfm
4. Remote control – not strictly necessary, but definitely helpful if you are installing the projector in a hard to reach place, or are using multiple projectors.
5. Interface method – the most common input methods for projectors include VGA, DVI and HDMI. Make sure your computer’s graphics card will allow you to easily connect to the projector. There are a variety of display type adapters out there, but that will increase cost and may (depending on the interfaces) result in loss of quality.

Step 8: Example #1: Single Quad Surface on Flat Object

Set up the scene

1. Turn on the projector, bring it into focus and position it so that the projected image is at least as big as the scene you want to map onto.
2. Mirror your computer's display so that it is shared with the projector.
3. Run SurfaceMapperGUI in Present mode (Ctrl + Shift + R).

Map the surface

1. If there are no quad surfaces on the screen, add a new one.
2. Drag the corners of the quad to match the corners of the actual surface being projected onto.
3. Use the Source file drop down to assign an image or video you want to the surface.
4. Switch to Render mode.

Notice that as the quad surface begins to match the geometry of the real object, all of the internal grid lines of the surface will become more and more parallel and evenly-spaced. This is a great way to tell when you're getting close!

Step 9: Example #2: Multiple Quad Surfaces on a Box

Example #2: multiple quad surfaces on a box

1. Set up the projector and open SurfaceMapperGUI as outlined in the previous step.
2. If there are no quad surfaces on the screen, add as many as you need.
3. Drag the corners of the quad to match the corners of the actual surface being projected onto.
   • Notice that when you move anchors points close to each other and release the mouse they will actually snap together. This can be very useful for mapping continuous surfaces or multi-sided objects with no edge artifacts.
4. Use the Source file drop down to assign images or videos to the surfaces.
5. Switch to Render mode.

Step 10: Example #3: Single Bezier Surface on Curved Object

Example #3: single bezier surface on curved object

1. Set up the projector and SurfaceMapperGUI as outlined in Step 8.
2. If there are no bezier surfaces on the screen, add one using the Create new bezier surface button.
3. Start by first positioning the unwarped bezier surface in the center of your curved object.
4. Adjust the four corners of the bezier surface to create the largest square you can within the curved object (don't worry about making the edges curved yet).
5. Use the smaller anchor points coming out of each corner to bend each edge to conform to the curve of the actual object.
   • You can also use the Horizontal force and Vertical force buttons to add uniform curving the edges.
6. Tweak the corners and small anchor points until the white border of the surface matches up with the edges of the physical object.
7. Use the Source file drop-down list to assign an image or video to the surface.
8. Switch to Render mode.

NOTE: You will quickly notice that as you bend the edges of the bezier surface they will appear very segmented and jagged. To fix this, hit the "+ Increase" button under Resolution until it looks nice and smooth!

Notice that as the bezier surface starts to match the physical geometry of the actual object, all of the dots will appear to be more and more evenly-spaced and fall along straight lines (well, arcs). This is a great indicator of how close you are to the actual geometry of the object!

Step 11: Example #4: Multiple Quads and Beziers on a Complex Scene

Example #4: multiple quads and beziers on a complex scene

1. Set up the projector and SurfaceMapperGUI as outlined in Step 8.
2. Add quads and beziers as needed using the buttons in the Program Options menu.
3. Position each of the surfaces onto the actual objects you want to map.
4. I recommend starting with all of the quads, then moving on to the beziers. Refer to the previous examples to see how to work with each of them.
5. Use the Source file drop-down lists to assign images or videos to the surfaces.
6. Switch to Render mode.

NOTE: Pay attention the layering of your objects. Start by mapping the objects that are the furthest from the projector (at the "bottom" of the stack). Surface re-ordering will be added eventually :)

Step 12: Example #5: Styrofoam Mannequin Head

While looking for simple squares and spheres for the other examples, I found some really great styrofoam heads at my local hobby craft store. I brought one home and used SurfaceMapperGUI to project a face onto it, and got it to look really nice after about 10 minutes of tweaking!

Example #5: styrofoam mannequin head

1. Set up the projector and SurfaceMapperGUI as outlined in Step 8.
2. Add a new bezier surface to the screen.
3. Position the surface near the middle of the head (maybe near the nose).
4. Use the Source file drop-down lists to assign images or videos to the surfaces.
5. Switch back and forth between Calibration mode and Render mode and adjust your bezier surface until the face matches up with the head.
   • Depending on the image you use as the texture you may need to distort the surface a little differently, but I found that having a roughly ovaloid shape with a flared bottom worked great.

Credit for the texture of the head goes to a user named "The ripper" on the Newtek forums. Image was found via Google Images (searched for "head texture", but comes from this forum thread.