CS766 Computer Vision, Fall 2007

Project 2: Panoramic Mosaic Stitching

Assigned: Sept 25, 2007

Due: Oct 10 noon, 2007

The instructor is extremely thankful to Prof Steve Seitz for allowing us to use this project which was developed in his Computer Vision class.

Project Description

Image stitching is a technique to combine a set of images into a larger image by registering, warping, resampling and blending them together. A popular application of image stitching is to create panoramas. Generally speaking, there are two classes of methods for image stitching: direct methods and feature-based methods. An example of direct methods is Szeliski and Shum's SIGGRAPH 1997 paper. Brown and Lowe's ICCV2003 paper, Recognising Panoramas, is a cool example for feature-based methods.

In this project, you will implement a feature-based method to generate panoramic images. You are expected to finish the following tasks to assemble panorama images.

1. Taking images. We prefer you to check out equipments (camera, tripod, and Kaidan head) in groups of two. Each group can check out multiple times, but should not hold the equipment for more than a half day for a single checkout. Pleaes contact TA for scheduling. Everyone is responsible to write all code of their own. Using the Kaidan head helps to place the camera optical center on the rotation axis. Here is a page describing how to use the setup. If you are using your own camera, you have to estimate its focal length (Brett Allen describes one creative way to measure rough focal length using just a book and a box, or alternatively use a camera calibration toolkit to get precise focal length and radial distortion coefficients). The parameters for the class cameras are given below. The following focal length is valid only if the camera is zoomed out all the way (wide angle).

The test images are provided for you to debug your code. If you want to calibrate a camera by yourself using the toolkit, you need to know Matlab a little bit. If you don't, here is a jump start.

2. Warping each image into cylindrical coordinates. Compute the inverse map to warp the image into the cylindrical coordinates as discussed in the class. You will have to use the focal length f estimates for the 480x640-resolution images provided above (you can either take pictures and save them in small files or save them in large files and reduce them afterwards) . If you use a different image size, do remember to scale f according to the image size.)

3. Computing the alignment of the images in pairs. You need to implement a feature-based translational motion estimation, using RANSAC method discussed in the class. We recommend you to use the SIFT features and you can download the SIFT program for this. This program does feature detection for you and saves the feature location and feature descriptors in a file. The README in the download is very self-explanatory on how  to use it. Given these features, your program should estimate a translational motion between each pair of images.

4. Stitching and cropping the resulting aligned images. Given the warped images and their relative displacements, figure out how large the final stitched image will be and their absolute displacements in the panorama. Then, resample each image to its final location and blend it with its neighbors. Try a simple feathering function as your weighting function (see mosaics lecture slide on "blending") (this is a simple 1-D version of the distance map described in [Szeliski & Shum]). For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences.

Crop the resulting image to make the left and right edges seam perfectly. The horizontal extent can be computed since the first image occurs at both the left and right end of the stitched sequence (draw the “cut” line halfway through this image). Use a linear warp to the mosaic to remove any vertical “drift” between the first and last image. This warp, of the form y' = y + ax, should transform the y coordinates of the mosaic such that the first image has the same y-coordinate on both the left and right end. Calculate the value of 'a' needed to perform this transformation.

5. Creating the final result. Convert your resulting image to a JPEG and paste it on a Web page along with code to run the interactive viewer. Click here for instructions on how to do this.

Bonus Points

Here is a list of suggestions for extending the program for extra credit. You are encouraged to come up with your own extensions. We're always interested in seeing new, unanticipated approaches and results!

• Sometimes, there exists exposure difference between images, which results in brightness fluctuation in the final mosaic. Try to get rid of this artifact. 
• Even if the exposure is consistent between images, vignetting often still creates brightness fluctuation in the final mosaics. Here is one solution to eliminate this artifact in panorama. You can try it out.
• Try shooting a sequence with some objects moving.  What did you do to remove “ghosted” versions of the objects? This paper suggested a solution, as briefly mentioned in our class.
• Make creative panoramas, like this one shown in our class.
• Implement a better blending technique, e.g., pyramid blending, poisson imaging blending,or graph cuts.
• How doe your code work on handheld image sequences? It probably depends on how precisely you can control your hands. If it doesn't work satisfactorily, try to use bundle adjustment to estimate motion.
• Try create panorama from an arbitrary set of images, i.e., a full implementation of AutoStitch!

You are welcome to do any other extensions or develop algorithm related to image mosaics. The bonus depends on how useful and difficult these extensions are.

Submission

• You have to turn in your complete source, the executable, the original pictures you have taken, at least one panorama image in addition to the one created from the test images, brief instructions to run your program, and a report in html format and hard copy put in TA's mailbox before the due time. Each panorama should be shown as (1) a low-res inlined image on the web page, (2) a link that you can click on to show the full-resolution .jpg file, AND (3) embedded in a viewer as described above. The report could contain a description of this project, what you have learned from this project, description of the algorithm you implemented, implementation details, what worked well and what didn’t. If you tried several variants or did something non-standard, please describe this as well. Please mention clearly what you have done for the extra credit so that we will not miss the opportunity to give you extra-credit. Note: you do not need to submit the library you used. However, you need to indicate which library you used and where you got it.
• You also have to indicate which is your favorite artifact generated by the program you have implemented (not the reference software). All class participates will vote for the top three artifacts. The creators for these artifacts will get extra bonus point.
• Please submit your result to the handin directory by the following:
  
  • First add the following line to your .cshrc.local file (located in your home directory):
    
    set path = ($path /s/handin/bin)
  
  
  • Then run the following command at the shell prompt:
    
    source ~/.cshrc.local
  
  
  • Finally,to handin your files, enter the following command
    
    handin -c cs766-1 -a PA2 -d directory
    
    where directory is the path to the directory where all your files are located.handin will go through the specified directory and hand in the tar ball. Please note that the handin script will copy everything in the directory and you should only put all files related to this project in that directory. Note : don't put the debug inforamtion in your source code.
    

Reference software

• SIFT: Scale Invariant Feature Transform
• AutoStitch: Automatic image stitching

References

1. R. Szeliski and H.-Y. Shum. Creating full view panoramic image mosaics and texture-mapped models, SIGGRAPH 1997, pp251-258.
2. M. Brown, D. G. Lowe, Recognising Panoramas, ICCV 2003.