In this assignment you are to take a sequence of images and create an image mosaic by registering, planar projective warping, resampling, and compositing them. First read the two papers:

• R. Szeliski, Video mosaics for virtual environments, IEEE Computer Graphics and Applications 16(2), 1996, 22-30.
• R. Szeliski and H.-Y. Shum, Creating full view panoramic image mosaics and environment maps, Proc. SIGGRAPH 97, 1997, 251-258.

The steps you should complete are:

1. Obtain an image sequence such as the one of Monona Terrace in /u/c/s/cs766-1/public/images/hw3/monona-terrace/ where the camera optical center does not change (much) during the entire sequence, and where each adjacent pair of images significantly overlaps one another. Other video sequences you may use are in /u/c/s/cs766-1/public/images/hw3/ You may use any of the provided sequences, but if you use a different sequence (either from the web or by using a camcorder) you'll get up to 10% extra credit. You should mosaic a minimum of four images together.

2. In order to automatically register the images, you'll use a feature point tracker called the KLT tracker. A local copy is in /u/c/s/cs766-1/public/mosaic/Klt/ Set the number of features to track as you see fit (at least 100). If you change any of the other parameters of the method, document these changes and justify your choices. Run this tracker with your image sequence to create a sequence of point correspondences. To learn about this tracker and how to use it, see the description at the KLT tracker web site. You should be able to modify the C program there called example4.c to do what you need. Note that in addition to creating a file of the tracked points, it also creates .ppm images showing the feature points detected in each image so you can verify visually that the method is working.

3. To compute the homography between each pair of adjacent frames, you must write a program that computes the forward or backward planar projective mapping associated with each candidate set of four point correspondences (i.e., four points in one frame and their corresponding four points in the second frame). To do this, read the following excerpt entitled "Projective Mappings for Image Warping" by Paul Heckbert for defining the necessary linear system of equations. Be careful to note that in the Heckbert paper the matrices are pre-multiplied by row vectors. (If, instead, you need to post-multiply column vectors, be sure to transpose the matrices first.) Heckbert's code for working with 3 by 3 projective transformation matrices is provided for you in the directory /u/c/s/cs766-1/public/mosaic/Heckbert/ Probably the easiest approach is to modify the code given in example4.c In particular, the C routine pmap_poly.c computes the planar projective transformation given four point correspondences. A C++ version called proj2d.cpp is provided, along with other necessary routines and a Makefile, in /u/c/s/cs766-1/public/mosaic/Heckbert/c++-version. The code includes a routine for solving an 8 by 8 linear system, though you could use Gaussian elimination instead if you desire (using Matlab or code from Numerical Recipes in C, for example).

4. Because the tracker is not perfect, one or more of the four point correspondences you use may be erroneous, so we need a way to determine when we have four good correspondences and, more to the point, a good homography. To do this, you must implement the RANSAC algorithm to find the homography that is consistent with the maximum number of point correspondences associated with a given pair of frames. The RANSAC algorithm is described on pages 346-348 in Section 15.5.2 of the textbook. You will use the RANSAC algorithm to test k candidate homographies based on k randomly selected sets of four correspondences (out of the n computed by the KLT tracker for this pair of frames). You should set the parameter k as well as the other parameters used by RANSAC empirically based on your experience using the algorithm. Document your parameter values used and explain your reasoning in choosing these values. Once you have computed a candidate homography based on four correspondences, compute the number of consistent correspondences from the remaining ones. Consistency will be based on a distance threshold of a new point from its position computed by the homography. Select the homography that results in the largest number of consistent points for each pair of consecutive frames. Finally, note that after finding the best four correspondences, the homography should ideally be recomputed using the four points and all of the other consistent correspondences that were found using least squares. You are not required to do this recomputation of the homography for this assignment, but if you do you will receive extra credit.

5. Warp each image into the projection associated with one of the frames (ideally one in the middle of the sequence) using the backward mapping method. Use bilinear interpolation to do pixel resampling. Compute the size of the output image by finding the minimum and maximum x and y coordinates.

6. Composite all the images using a feathering algorithm that uses a bilinear weighting function for all of the pixels contributing at a given point. This method is described in Equation (9) in the paper by R. Szeliski, Video mosaics for virtual environments, IEEE Computer Graphics and Applications 16(2), 1996, 22-30. Your result image should not show noticeable seams.

Reminder

1. The iteration needed for RANSAC is over 10000
2. Don't debug the entire program at once. You should at least chop your debugging into the following process
• Read the manual of KLT in Select and Track features and their related parameters
• Try example3 to make sure that you know how to use the program and how the data is stored in the program
• Test the homography program in Heckbert's library.
• Test RANSAC
• Test blending two images

1. Linux version: SourceCode
• Download the source code
• Unzipped it by tar -zxvf hw3.tgz
• Go to the directory by cd HW3
• Create the dependency by make depend
• Create the main program by make or
• Create the example of KLT in Klt directory by make exampleN where N from 1 to 5
2. Window version: Source Code
• Download the source code
• Unzipped hw3.zip
• Double click on the solution file HW3.sln
• Rebuild the entire solution
• All the libraries and all the example executive files are created in the directory, dlls.
• You can put your test images in the dll directory and run the debug of the desired example.

• Hard copy
1. A description of the methods you implemented for accomplishing each of the above steps,
2. A description of the parameter values used and how you selected them for the KLT tracker and the RANSAC algorithm
3. Results showing at least two of your input images and the output mosaic image.
• Handin directory
1. Source code of all your programs
2. All the source images

Extra Credit
Here are some ideas for extra credit. The amount of extra credit you receive will be based on the additional code you write, the writeup of what you did, and the extra results you produce. If you have other extensions you'd like to try, those will be considered extra credit as well.

• Take your own image sequence and use it as data for testing your method.
• Instead of the hat-shaped blending function used in Step 6, implement Burt and Adelson's pyramid blending method discussed in class and described in their paper A multiresolution spline with application to image mosaics, ACM Transactions on Graphics 2, 1983, 217-236.
• Instead of assuming the set of input images is a video sequence in which successive images are very similar, allow successive images to be only partially overlapping. In order to do this, you'd first have to extract the corner detector that is in the KLT tracker (or else implement your own), and throw away the rest of the KLT tracker. Next, you could either do a brute force RANSAC search for four good point correspondences, or else you could implement Gaussian pyramids, and then do a coarse-to-fine search for the four point correspondences.
• Implement a different method for video mosaicing such as one of those described by Smolic and Wiegand or by Rousso et al. or by Gonzalez et al. or by Irani et al. Do a web search on "video mosaic" to find other methods.