I took the images using the "Canon SX100 IS, tag 6418204056". I took all the pictures completely zoomed out and using the default resolution. I also used manual mode to take the pictures.
For warping to the cylinder co-ordinates, I used the inverse warping equation given in the lecture notes.[Actually the equations have to be combined with the equations for removing radial distortion.]. Since my pictures were taken with the default zoom and resolution, I used the parameters given on the course webpage.
After warping to the cylindrical co-ordinates, there are regions in the image that are empty. So I also maintain a mask image with 0 bits wherever the image has no pixels mapped and 1 elsewhere. This mask is then later used for setting the weights for each pixel.
- Feature Extraction - I use the SIFT tool to extract the features from each image.
- Matching Images - Here I assume that every image overlaps with the next image in the sequence. Hence, I just find the matching features between consecutive images. For finding the matching features, I take the SIFT identifiers from one image and find identifiers with SSD between id's less than threshold(10) and the ratio of best and second best SSD is less than epsilon(0.6). The threshold may not be enough but I realized, its better to rely on RANSAC to remove the mismatch.
- RANSAC - Since we are assuming only translation motion there are only two unknowns, which can be computed with just one matching point. But to be a bit on the safe side, I randomly selected two points and used least square method to determine the translation matrix. The number of iterations were computed using the probability of inliers = 0.4 and probability of finding the right H at 0.99. The translation matrix with the highest inliers were selected.
Now that we know the pairwise translation matrices and since these operations can be composed, I multiply the translation matrices for all the images before the current image to get the matrices to translate all images into the space of the first image. The first image has an identity translation matrix.
To compute the shear, I find the position where (0,0) from last image is mapped to in the new space, say (xm, ym). The last image is the same as the first image and would be ignored in the final mosaic. So its (0,0) should have the same y value as the first image. So shear ,a = -ym/xm.[Actually (1-ym/xm) since I used Matlab where (1,1) is the first point].
Also, I compute the smallest x and y values taken by any image in the new space and shift the origin to that point so that all images are mapped to the positive xy co-ordinates.
We multiply the shear and origin shifting matrices into the translation matrices.
Instead of computing the weights for each pixel, we compute the weight for every column. We use a linear function for the overlapping regions such that the weight is 0 at the edge. It seems my photos had excessive overlap and there is a possibility for a small region to have three overlapping images. So I had to normalize the weights such that they summed up to 1 for all images.
Since I normalize the weights, I can just multiply the image by the weight map which is the same size as the image. Perform the forward transform using matlab's imtransform and then just add up the resulting images in the new space. Just as a side note, I was initially computing inverse warping co-ordinates for each pixel and computing the weights for each. This turned out to be very slow and took more than an hour for every image. By using the new approach of weight maps included in the masks, I am now able to run the code within 5 minutes.
I had a set of images that had moving cars which would result in ghosts in the images. Hence, I implemented the ghost removal approach from the paper [2]. The given paper creates a graph of the mismatching regions and ignores the values from images that form a vertex cover in the graph. It also weighs each image based on its position so that it uses the pixels from images where the mismatched part is nearer to the center. I also take into account whether the mismatching pixels are completely subsumed in one particular image and boost its weight. To obtain chunks of mismatched columns rather than many small stretch of columns, I reduce the threshold everytime I see one mismatched column and increase it again if the columns start matching.
They also try to fix the difference in exposure, but since I used all images with manual exposure, I could skip this step.
The input image sets are in Mosaic/set* folders.
The output images are in Mosaic/ folders.
To run the code, in Matlab run mosaic.m. Modify the directory name in mosaic.m for different sets.
To enable deghosting, set remove_ghosts=1 in mosaic.m.
Low Res photos marked with _small.jpg
This is my favourite artifact[File: Mosaic/set2_new2.jpg]
Result on the input set. Since I didnt use multiband combining, the final image has dark and light regions.
This is the result with hand held images. As you can see there are portions that are missed and hence black.
This is the result without removing ghosts. As you can see there is a ghost bus in the picture.
This is after removing the ghosts. As you can see, since we pick every column for one image, the final image is not smooth because of every mismatch in the columns comes from different images that may have slight exposure differences.
