This project implemented Szeliski and Shum's panorama stitching technique. Images were cylindrically warped, homography between images were found, and then the images were stitched together into a complete panorama. Michael completed the warping, shearing, and blending code while Sajika worked on the code to interface with SIFT, determine shear, and load in, process, and output the resulting panorama. Both collaborated on picture taking and bug fixing.
The project was completed in Matlab, using the SIFT bindings to find homographies but no other exterior libraries. While the warping, blending, and matching are very fast, finding SIFT homographies and running RANSAC take order 10^0 seconds per image. The main function is createPanorama(indir,outfilename,f,is360,isfullhomography), where indir is the directory of pictures (ends in /), outfilename is the name of the final image, f is the focal length (ours was estimated manually using the two book method described in class, ~503 pixels for a 640x480 picture), is360 is a boolean about whether to blend the first and the last images together, and isfullhomography is whether to use just cylindrical warping, or to estimate rotations and perspective between frames using SIFT.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Sajika experimented with using SIFT to solve for the full homographies on sets of unwarped images, rather than simply translation of warped images. This method is more robust to camera movement, but results in some odd artifacts and difficulty in performing the blending. Here is an example of how the full homography solver can correct for rotations:
|
|
|
|
|
Michael experimented with using variable window linear feathering, e.g. if there are x overlapping pixels in a row or column, then the alpha channel takes x pixels to transition from 1 to 0. The resulting images are far smoother and seamless than a simple even alpha weighting scheme. Observe the differences in the railing, and on the seams (floating point errors may result in extra lines around the mask of the unfeathered images):
|
|
|
|
|
We performed some experiments with non-steady images, as well as situations where the traditional algorithm might have difficulty.
A gallery in the Chazen
Museum. Note that tripods are forbidden in Chazen, so the images were taken by hand.
The algortihm was robust enough to still produce reasonably good results (although
notice the ghosting in the section of the image with the the piano).Interactive
The docks by Memorial
Union. We picked this shot because there are both large numbers of moving objects
and wide featureless stretches of water, as a challenge to the SIFT algorithm. The
results have a lot of ghosting but are still not too bad. Oddly enough, vignetting
turns out be the most prevalent here in the outdoor scene framed by the setting
sun. Interactive
Bike Rack near Memorial
Union. Note that the horseshoe shaped rack is almost a straight line in the image.Interactive
In front of Engineering
Hall.Interactive
A residential intersection.Interactive
The pedestrian bridge
on N. Park St. Due to the time lag between images, the same man appears in two images,
both arriving and leaving. Also note that there is ghosting when the railing gets
"too close" to the camera for our warping assumptions to hold. A true 360 panorama.
Interactive
The front of the Computer
Science Building. A true 360 panorama. Interactive
Camp Randall Stadium.
Interactive
Most of the portions of the project went well. The majority of the imperfections in our image were due to ghosting. Where ghosting artifacts occurred were mostly where we expected, e.g. where objects were very close to the camera, in motion, or the tripod and camera were not as steady as possible. We had some issues creating full 360 panoramas: blending the last and first images together seemed to create a lot of artifacts.
We learned quite a bit about using Matlab to increase the performance of affine transformations as well as arbitrary warps. It was also a learning experience to see which images "worked" and which did not. At the end we had a good appreciation of the computational difficulties in matching and stitching images.