- Taking Images
I took the images using the "Canon SX100 IS, tag 6418204056". I took
all the pictures completely zoomed out and using the default
resolution. I also used manual mode to take the pictures.
- Warping to cylinder
For warping to the cylinder co-ordinates, I used the inverse warping
equation given in the lecture notes.[Actually the equations have to be
combined with the equations for removing radial distortion.]. Since my
pictures were taken with the default zoom and resolution, I used the
parameters given on the course webpage.
After warping to the cylindrical co-ordinates, there are regions in
the image that are empty. So I also maintain a mask image with 0 bits
wherever the image has no pixels mapped and 1 elsewhere. This mask is
then later used for setting the weights for each pixel.
- Computing Alignment
- Feature Extraction - I use the SIFT tool to extract the features from each image.
- Matching Images - Here I assume that every image overlaps with
the next image in the sequence. Hence, I just find the matching
features between consecutive images. For finding the matching
features, I take the SIFT identifiers from one image and find
identifiers with SSD between id's less than threshold(10) and the
ratio of best and second best SSD is less than epsilon(0.6). The
threshold may not be enough but I realized, its better to rely on
RANSAC to remove the mismatch.
- RANSAC - Since we are assuming only translation motion there are
only two unknowns, which can be computed with just one matching
point. But to be a bit on the safe side, I randomly selected two
points and used least square method to determine the translation
matrix. The number of iterations were computed using the probability
of inliers = 0.4 and probability of finding the right H at 0.99. The
translation matrix with the highest inliers were selected.
- Computing shear
Now that we know the pairwise translation matrices and since these
operations can be composed, I multiply the translation matrices for
all the images before the current image to get the matrices to
translate all images into the space of the first image. The first
image has an identity translation matrix.
To compute the shear, I find the position where (0,0) from last image
is mapped to in the new space, say (xm, ym). The last image is the
same as the first image and would be ignored in the final mosaic. So
its (0,0) should have the same y value as the first image. So shear ,a
= -ym/xm.[Actually (1-ym/xm) since I used Matlab where (1,1) is the
Also, I compute the smallest x and y values taken by any image in the
new space and shift the origin to that point so that all images are
mapped to the positive xy co-ordinates.
We multiply the shear and origin shifting matrices into the
- Computing Weight Maps
Instead of computing the weights for each pixel, we compute the weight
for every column. We use a linear function for the overlapping regions
such that the weight is 0 at the edge. It seems my photos had
excessive overlap and there is a possibility for a small region to
have three overlapping images. So I had to normalize the weights such
that they summed up to 1 for all images.
- Combining the images
Since I normalize the weights, I can just multiply the image by the
weight map which is the same size as the image. Perform the forward
transform using matlab's imtransform and then just add up the
resulting images in the new space. Just as a side note, I was
initially computing inverse warping co-ordinates for each pixel and
computing the weights for each. This turned out to be very slow and
took more than an hour for every image. By using the new approach of
weight maps included in the masks, I am now able to run the code
within 5 minutes.
- Removing ghosts : EXTRA
I had a set of images that had moving cars which would result in
ghosts in the images. Hence, I implemented the ghost removal approach
from the paper . The given paper creates a graph of the mismatching
regions and ignores the values from images that form a vertex cover in
the graph. It also weighs each image based on its position so that it
uses the pixels from images where the mismatched part is nearer to the
center. I also take into account whether the mismatching pixels are
completely subsumed in one particular image and boost its weight. To
obtain chunks of mismatched columns rather than many small stretch of columns, I
reduce the threshold everytime I see one mismatched column and
increase it again if the columns start matching.
also try to fix the difference in exposure, but since I used all
images with manual exposure, I could skip this step.
The input image sets are in Mosaic/set* folders.
The output images are in Mosaic/ folders.
To run the code, in Matlab run mosaic.m. Modify the directory name in
mosaic.m for different sets.
To enable deghosting, set remove_ghosts=1 in mosaic.m.
Low Res photos marked with _small.jpg
This is my favourite artifact[File: Mosaic/set2_new2.jpg]
Result on the input set. Since I didnt use multiband combining, the
final image has dark and light regions.
This is the result with hand held images. As you can see there are
portions that are missed and hence black.
This is the result without removing ghosts. As you can see there is a
ghost bus in the picture.
This is after removing the ghosts. As you can see, since we pick every
column for one image, the final image is not smooth because of every
mismatch in the columns comes from different images that may have
slight exposure differences.
1. Construction of panoramic mosaics with global and local alignment - Heung-Yeung Shum and Richard Szeliski.
2. Eliminating Ghosting and Exposure Artifacts in Image Mosaics - M. Uyttendaele, A. Eden, and R. Szeliski.