No-Stitch Panorama Picture

Intro: No-Stitch Panorama Picture

Hugely wide panorama picture kept fascinating me since I was a kid. Photo stitching software seem to have done a lot of progress these days. However, I was intimidated by the process of shooting the pictures (slightly overlapping, horizontally aligned) and doing the manual stitching adjustments, etc. and never made the step(1).

An alternate method, based on capturing a rotating vertical scanline, and then letting some software do all the job, has been in my head since about two years.

Having received a whole set of Fischertechnik (2) components helped me decide to experiment it, just for the sake of verifying the idea.

Then, although the whole is still a work in progress, the results are so good that I wanted to share them with you. It has room for many improvements to increase resolution and usability, but it basically works pretty well.

Read on, make one, and show your results!

(1) In short, I have much more interest in spending time engineering something (that can later be effortlessly reused), rather than tediously executing manual work.
(2) Fischertechnik is to mechanic what breadboard is to electronic --Well it's more than just that, but you get the idea.

STEP 1: The Idea

The principle
A movie consists of a succession of pictures shot rapidly (30 per second). We will only consider the vertical central line (1 pixel thick), that we'll call the scanline. When slowly rotating the camera, each frame will be like the last one, but very slightly shifted. So the scanline will sweep the whole scene.

Being in the center of the frame, the scanline will have no lateral barrel distortion. The scanline will also be very well horizontally aligned (relative to the rotation axis).

In order to collect more vertical pixels (i.e. to have a taller scanline, hence a higher vertical resolution), the camera is rotated by 90°. This rotation will have to be taken into account during the post-processing.

In summary, there are two phases:

1. Capture: The first part consists of shooting the movie in slow and continuous rotation. It only needs a camera able to shoot video clips, and a motorized rotating mount.
2. Post-processing: The second part needs a computer program. It consists of extracting the central column of pixels (corresponding to the scanline) from each movie frame, and stitching them together to form the panorama image.

Enough theory, now let's test this principle

So if you want to experiment too, you must be (for now) familiar with computer command-line tools:

• the installation of command-line tools,
• the operation of command-line tools,
• the edition of script files, to tweak some parameters.

I will only provide instructions for experienced such users, and not how to get acquainted to the command line.
Tested on Linux (Ububtu) and Mac OSX so far.

But, upon popular request, I may write a user-friendly program that anybody can user.

STEP 2: The Questions

1. What rotation speed and precision is required, and can it be reached with toy motors?
2. Is the quality and resolution of a video clip sufficient?
3. What kind of camera is required?
4. What computer power is necessary?
5. Can it be achieved with open-source software?
6. What is the relationship between the final image resolution and the lens aperture, the movie frame-per-second, and the rotation speed?
7. What is the effect of exposure adjusts during the recording?
8. How will moving objects appear?
9. What kind of perspective distortions will appear?

Answers in the last step. So please read on...

STEP 3: The Rotating Mount

There is nothing complex with this mount:

• Red plate: the fixed part, four feet (for use on a table), a fixture for a tripod, a battery holder.
• A spirit level
• The motor with gears, rotating the turret
• The rotating turret, consisting of a big gear and a chair for the camera

The only tricky part is to obtain a very slow rotation (approx one round in two minutes or more).

The depicted setup gives one round in one minute:

Later, I added more gears. Now one round takes 3 minutes, but it is less reliable. The definitive version will use a servo modified for continuous rotation.

STEP 4: The Software

As said, this is still very experimental, and meant for power users who know how to operate the command-line and edit text files.

For the other more casual users, wait for a user-friendly version of the program, that I may write soon.

If you want to try the experimental way, read on. You need four things:

1. FFMPEG

• On Mac OSX: first install XCode (from your "Mac OSX Install DVD"), then svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg, then read the INSTALL file (./configure && make && sudo make install).
• On Debian/Ubuntu Linux: sudo apt-get install ffmpeg do the same as for Mac OSX, because you need a more recent version than what apt-get provides.
• On Windows: follow the "FFmpeg Windows Builds" on http://www.ffmpeg.org/download.html . Be sure to add the path of the binaries directory to your PATH environment variable.
• Then, on any platform, when done, check from a shell (aka terminal on OSX, aka CMD or DOS window on Windows):
  type: ffmpeg -version
  and it should say something like:
  FFmpeg version SVN-r26402, Copyright (c) 2000-2011 the FFmpeg developers
  built on Apr 22 2011 09:07:41 with gcc 4.2.1 (Apple Inc. build 5664)
  etc.

2. ImageMagick

• On Debian/Ubuntu Linux: sudo apt-get install imagemagick.
• On Mac OSX, and Windows: follow the corresponding page under "Binary Releases".
• Then, on Windows, be sure to add the path of the binaries directory to your PATH environment variable.
• Then, on Mac OSX, edit /home/Laxap/.profile and add:
  export MAGICK_HOME=/Users/Laxap/Downloads/ImageMagick-6.6.7
  export PATH="$MAGICK_HOME/bin:$PATH"
  export DYLD_LIBRARY_PATH="$MAGICK_HOME/lib/"
  Replacing Laxap by your username. Also replace .../Downloads/... by the location where you decompressed the ImageMagick archive.
• On any platform, when done, check from a shell:
  type: convert -version
  and it should say something like:
  Version: ImageMagick 6.5.7-8 2010-12-02 Q16 http://www.imagemagick.org
  Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC
  etc.

3. Python

• On Linux (Debian and Ubuntu) and Mac OSX, Python comes already installed because many system utilities use it, so there is nothing to do.
• On Windows, install it, then be sure to add the path of the binaries directory to your PATH environment variable.
• On any platform, when done, check from a shell:
  type: python --version
  and it should say something like:
  Python 2.6.1

4. My script:

• Download the script panoramize.py attached to this step, and save/move it to a wanted location.
• On Linux and Mac OSX, make it executable: chmod +x panoramize.py

STEP 5: Shooting a Movie

Find a good place.

Install tripod, attach mount on it.

Tune tripod to be lever.

Set camera on movie mode. Place camera on the rotating mount. Check every mechanical parts. Test rotation.

Turn motor on. Press camera trigger.

Wait for one complete rotation, plus some margin. Stay behind camera.

Stop camera. Stop motor.

Inspect movie for proper sharpness, exposure, and scene scrolling.

STEP 6: Running the Software

(Like in the past, it is thrilling to come back home, work a bit, and see the results. Or you could take a good netbook on the go.)

Transfer your movies to the computer.

Open a shell and type:
./panoramize.py IMG_12345.AVI

You can specify more than one file as command argument.

You may have to add "python" in front:
python panoramize.py IMG_12345.AVI

To adjust parameters, edit the panoramize.py file, and change:

• ROTATE
  0: landscape, 90: portrait. Acc. to camera orientation.
• RATIO
  e.g. "80%", acc. to lens aperture and rotation speed
• MIRROR
  True or False. Acc. to camera and rotation direction

ROTATE and MIRROR only depend on the construction of your mount.

RATIO depends on the rotation speed, but also on the view angle (i.e. the focal distance) of the zoom value that you are using. If you use the same default zoom factor, you won't need to change the value once tuned.

STEP 7: First Results

Landscape panorama just look like... panoramas!

• First image: A collage of three panorama images. See a bigger version. In reality these images are approx 4,000 pixel wide!

Indoor panoramas give a very unusual view of familiar places.

• Second image: The object being close, the perspective distortion effect is very strong.
• Third image: Cropping and keeping only the middle half will eliminate the strongest distortion areas and give a more natural aspect.

The only problems I could see are the vertical dark/light bands due to readjustment of the exposure.

When trying to have people on the picture: it's hard to be really still. Moving will yield strange or funny effects. Try it during parties!

STEP 8: Answers

Answers to the questions posed in Step 2

1. What rotation speed and precision is required, and can it be reached with toy motors?

• Speed strongly depends on the video resolution, FPS, and focal distance. Approx one round in 2-3 minutes for 640 pixels, 30 fps, f 2.8.
• Toy motors are OK but require considerable gear ratio to achieve this low rotation speed. This will result in noisy mechanics, but with the advantage of a considerable torque with inexpensive motors. The gears should work reliably, else distortions will be generated. Servo modified for continuous rotation may give the best results in all respects.
• With battery or accumulator power, the rotation speed might not be extremely precise, but sufficiently stable.

2. Is the quality and resolution of a video clip sufficient?

• Yes it is largely sufficient as long as you target web resolutions. For poster prints, the resolution will not be sufficient.

3. What kind of camera is required?

• Cheap cameras are sufficient. I used a Canon Digital IXUS 95 IS. In fact it is quite ideal due to its compactness.

4. What computer power is necessary?

• With a 2.4 GHz Intel dual core, a 4423 x 512 panorama takes 3 minutes to compute. This can be shortened by a better Python script.

5. Can it be achieved with open-source software?

• Yes, in fact very easily, at very high quality. ffmpeg is blazing fast at converting MJPEG into indifidual JPEGs, and imagemagick is ideal for cropping/stitching. Python is great at orchestrating the whole (but for this many other languages would fit). They have ports for many major platforms.

6. What is the relationship between the final image resolution and the lens aperture, the movie frame-per-second, and the rotation speed?

• What matters is that the final pixels represent a view angle that is the same in both vertical and horizontal direction.
• In vertical direction, i.e. along the scanline, the view angle is determined by the F-number of the zoom value, and the movie pixel resolution.
• In horizontal direction, the view angle is determined by the movie FPS, and the mount rotation speed.

7. What is the effect of exposure adjusts during the recording?

• This really depends on your camera, and how abruptly/smoothly the adjust is performed. But it generates vertical bands of different exposures. This is a bit problematic. Can some cameras be told not to readjust the exposure during a movie? Maybe some cheap/old ones?

8. How will moving objects appear?

• As long as they do not cross the scanline, moving objects will not be seen (unless they are near and affect the overall scene luminosity).
• Fast moving object will appear as a single column of pixels.
• An object moving around the camera in sync with its rotation will appear as an horizontal band.
• Objects moving "at reasonable speed" in the same direction as the camera rotation will appear thicker, and in the opposite direction, thinner.

9. What kind of perspective distortions will appear?

• Vertical objects will have no distortion at all, because they correspond to the scanline, which is at the center of each frame, having no lateral distortion.
• Horizontal objects will have fish-eye-like distortion. Horizontal parallel straight lines will appear as hyperbolic lines, meeting at a vanishing point on the horizon line.
• In life, although we constantly see them, we no longer notice the distortions because (1) when checking the straightness of big objects we tend to move our eyes to focus along straight lines and (2) our brains compensates and removes the distortions.

STEP 9: Conclusion and Next Steps

Conclusion

This little experiment has proved to be a more than viable solution. It is easy to obtain good results. It can be combined to pole camera mount, for aerial panorama, or anything else you can imagine. The strong perspective distortion can either be used as a wanted effect, or removed by cropping to keep the central 50% of the image height.

There are numerous possibilities for building a good mount, including laser cutting of plywood or acrylic.

Instead of a camera, use the lens of an old reflex camera and the sensor of a scanner. You should get insane resolutions! Rotation must be very very slow.

Thanks for reading!

Next steps

• Slower rotation, usage of a servo modified for continuous rotation.
• Usage of a camera with higher resolution.
• More compact and robust mount.
• Microcomputer (e.g. Arduino) to ease operations: return to given position, motor and movie start/stop sequence; remote control for a one-button operation.
• Guessing the ideal rotation speed, given the resolution, FPS and focal distance.
• A user-friendly program with GUI, to compute the final image and adjust parameters (on top of the python script) -- see 2nd picture of this step.

Acknowledgments

I must warmly thank Udo, my wife's Uncle, who donated us more than 20 Kg of Fishertechnik. As a teacher, he was using it at school; now he's retired and we're the happy recipients of this awesome technology!

Comments

[shyamvnarayanan]

Hi there,

My goal is to make a panoromic image of a moving object. The camera stays still in one spot and takes a bunch of images of a large object that moves past it. My goal is to stitch these images together and make a high res image of the moving object. Is it possible to use your tutorial above to accomplish this ? So for instance make an .AVI file of the moving object first and then feed it to the panaromize.py script. Would that work ? If no, could you suggest an alternate approach ?

[laxap]

Hi Shyamvnarayanan,

I would need some more details on your goal, like:
- what could be the moving object,
- why do you want to have it in a panoramic picture,
- is the fact that it is moving an issue, or a feature, etc.

For example: "I want to make a panorama of a landscape, with a car moving and leaving a motion blur trail on 1/4th of the image". Or the same, but the car shall be sharp. Or "I want to make a panorama of a crowded downtown location; the people shall be frozen / the fast moving people shall be motion-blurred". Etc.

Greetings,
Laxap

[shyamvnarayanan]

The moving object is a train. My goal is to take a panoramic image of the entire train. The moving is a feature. My camera is stationary and wont move. If I can take a collection of pictures of the train - say for example of each carriage as it passes by and then stitch them together into a panoramic image I would achieve my desired goal. We can assume that the train is moving very slowly - say under 5mph.

[Light_Lab]

Nice pictures but I am a bit confused; in the heading you say "No-Stitch..." yet in the description you say ".....and stitching them together to form the panorama image".
I really can't see how this is different to how "Panorama Maker" makes panoramas out of video files except you are using a motorized rotating mount. I guess the mount could do a better job than hand held, but at 25 to 60 frames a second there is already a lot of image redundancy. Most video-panoramas I have made work quite well with the camera waving about all over the place.

[BrianV8]

Try the panaram maker with a verticle video. It works great horizontally but verticle the function colllapses. It is searching for horizontal ques such as the horizon etc. This guy is just scanning.

[Light_Lab]

Panorama Maker (like all Panorama software I have seen) uses point cues that work perfectly for both vertical and horizontal panoramas even tiled combinations of both axises. It makes excellent panoramas from all slowly panned videos in all directions:

[Light_Lab]

I forgot to mention this was shot hand held and I deliberately gave it a twist at the start to show how robust the image stiching algorithm is and how redundant smooth synchronisation is with state of the art panorama software.
BTW If you think Panorama maker is too expensive you can do it for free with a little extra effort using Hugin:
http://hugin.sourceforge.net/
and feed it with images from your video with a free video to JPG converter like:
https://www.dvdvideosoft.com/products/dvd/Free-Video-to-JPG-Converter.htm#.VbC6L_nIE5w
Hugin is a remarkable freebie that can assemble a number of 2D panoramas from a folder of related images. Often with little or no user tweaking.

[BrianV8]

I am developing a verticle panaramic program but am using the audio channel to input a pulse. I suggest this versus a mechanism at constant speed. Try some sort of encoder and a low voltage source. In your code, listen for the pulse, grab the frame, strip out the column of pixels, stack them together and there you have it. Im actually not doing a panaramic but a very long image of a road or path as one continuous image that is in focus for the mile of path. Good luck all. Im building it in matlab but would love a new option from others that can code this another way. Might pay too.

[timweaver17]

this is excellent! just wondering if you have any sort of formula that could be used to calculate various variables, like everything listed in bullet 6 on step 8...

[laxap]

What I do to calibrate the vertical ratio (see pic 2 in step 9) is manage to have a spherical object in the scene, then adjust the ratio to get the same diameter in all directions. This ratio will be valid for a certain zoom, resolution, and rotation speed.

This empirical ratio could tune a quite simple formula... I'll keep you updated.

[tictockrock]

I love panoramas since I discovered the function on my camera.
It takes 3 pictures, after each one the side of the last one is shown so that you can line them up. It gives very good results (with some funny errors sometimes).
The problem with this is that people and objects need to keep perfectly still for it to work.
I would love to have a program that does what you described!

[ogorir]

something that might be worthwhile looking into with this setup is a newer DSLR that shoots 1080P 60fps. you can lock your exposure and focus so you don't get the banding and could rotate the camera faster, not to mention 1080 px image height.

[NaturalCrafter]

You enspired me. The most I have done was stitch photos with picture publisher software, but as you mentioned the two photos your trying to stitch don't always match up just right. I did have a method of side stepping two ft and then photographing what ever was in front of me again or turning and loosing half a photograph with each shot. I will look through my photos using your technique. Hope I earn a patch. Take a look at my sock maker when you have a minute.

[laxap]

Are you intending to do a movie-to-picture panorama (as described in this ible)?  Tell me if you need help with converting the movie.

To get a patch, post either a panorama made this way, or your mount (or a work in progress).

[frogg_butter]

Windows Live Photo Editor also has a really nice panorama feature. Just browse to the photos you want to stich and it does pretty much all the rest.

[rapidprototyping]

maybe a rotation table from an old microwave oven might work for turntable.
and I recently bought a HD mini DV camera RD52 SHE 2 MP camera mini SD card with 2GB card 640 X 480 pixels 30 framesper second resolution 1024 X 960 DPI video plays in media player no conversion required. camera cost less the fourty dollars on hobby parts web site hope this helps (Rapid)

[HMBeck]

You have an extremely nice looking house and property. It looks amazing!

[MaJonesy]

I love the concept and the outcome, but I feel you might of over complicated shooting Panorama's. I know there are millions of ways to shoot them, with good and bad results.

The method I use to shoot my panorama's is very simple and can get really good results with minimal effort. Just use any camera, place it on a tripod take as many shots as you can (I usually end up taking about a hundred or more)... Then upload all your shots to your computer and use a free program called "Microsoft ICE" (Image Composite Editor)... The program will automatically stitch it all together, after this you can crop the image to any size you want and export it to many different formats..

Program Overview Page: http://research.microsoft.com/en-us/um/redmond/groups/ivm/ICE/

[rimar2000]

Excellent!