CS766 Project 3: Photometric Stereo

Project Report

Aubrey Barnard

Monday, November 03, 2008

Description

Photometric stereo is recovering an object's three-dimensional structure from a series of images where the camera location does not change but the lighting location does change. The general idea is that, based on the highlights and shadows of the object, you can recover its shape, similar to how humans perceive shape.

I used Matlab to create a program to recover an object's structure from a series of images. There are five basic steps.

  1. Find the lighting direction in each image based on an image of a chrome sphere with the same lighting direction.
  2. Based on the series of images and their lighting directions, find the normal of each pixel on the object's surface.
  3. Use the normal map to find the albedo map.
  4. Use the normal map to find the depth map.
  5. Save the results.

My program appears to successfully carry out these steps to recover the structure of the object. You can view the results below, and also read about how to use my program, some specifics of my implementation, and some significant learning outcomes.

About the Software

Inputs

There are four inputs to my program. They are:

  1. a cell array of the file names of the object images
  2. a cell array of the file names of the chrome ball images
  3. the file name of the object mask image
  4. the file name of the chrome ball mask image

As a convenience, I also made a Matlab function that takes the name of one of the experimental image sets ('gray', 'buddha', 'horse', 'cat', 'owl', 'rock') and creates the four arguments for the main function. This function, 'generateTestImagesFileNames', makes many assumptions, and is intended to only work with my project setup.

Outputs

The main function returns the albedo map, the normal map, and the depth map. This allows the function to be used interactively at the Matlab prompt. The main function also saves the following artifacts as images: the albedo map as 'albedoMap.png', two styles of normal map as 'normalMapRegular.png' and 'normalMapPsychedelic.png', and five views of the reconstructed surface as 'surface*.png'. The views are from directly in front of the object, from above, from left, from below, and from right. All the output images are stored relative to the current working directory.

Invocation

I recommend the following method of running my program. Run Matlab from the base directory of the project. Once Matlab has loaded, run the following command:

[objs chrs objm chrm] = generateTestImagesFileNames('gray');

This generates the list of file names of the object images, the list of file names of the chrome ball images, the file name of the object mask image, and the file name of the chrome ball mask image. (This only works for my project setup. You will have to make your own lists of file names for any other setup.) Then pass these file names to the main function, thus:

[amp nmp dmp] = main(objs, chrs, objm, chrm);

This generates the albedo map, the normal map, and the depth map as well as saving the results as images.

Results

Here are the results of my program working on the experimental images. The thumbnail images link to full-size images.

Image SetAlbedo MapDepth MapNormal Map 1Normal Map 2Front ViewAbove ViewLeft ViewBelow ViewRight View
Buddha
Cat
Gray
Horse
Owl
Rock

The results speak for themselves pretty well. All of the reconstructions were successful, but with varying artifacts from noise. Clearly there can be some improvement. I would first try eliminating more noise, but I would have to balance that against smoothing the reconstruction too much. It would be nice to be able to recover more of the object, but that would probably require multiple camera angles, which is beyond the limitations of this method.

I also feel like the recovered depth maps have a greater depth range than the actual objects. (That is, the recovered depths are exaggerated.) Addressing this issue would help the 3D surfaces look much more realistic. Unfortunately, I am not sure how to accomplish this, other than to use a different approach to reconstructing the surface. Perhaps, I could construct an alternate system of equations that would better capture the depth constraints such as the distance from the object to the background. It might also be possible to investigate other solutions to the system of equations, and pick a different solution that "looks more realistic", e.g. has a smaller depth range. Overall, exaggerated depths may be a limitation of this approach, and a different approach altogether would be needed to address this problem.

Implementation Details

I have made a few significant implementation choices completing this project. The first significant aspect of my implementation is that I convert all the images to black and white prior to computing the lighting directions. This has the advantage of reducing computational work and can lessen the effects of noise. Furthermore, textural information is best represented by black and white, as is done in the CIELAB color space.

Another significant implementation aspect is that I only use the pixels in the object for the depth map reconstruction. This approach has the large benefit of greatly reducing the size of the system of linear equations used to solve for the depth map (both the number of coefficients and the number of variables). The result is increased efficiency and ability to deal with larger problems. The tradeoff is that this approach requires a map the size of the number of object pixels. Even with the additional memory for the map, this approach uses less memory than a system of equations involving all the pixels (at least for the given experimental images where the object is only about one third of the image).

Specifically, my approach finds the pixels that are part of the object from the object mask. Matlab orders these pixels linearly, but retains the original location of each pixel. Since we need to know the locations of the neighboring pixels to assemble the system of equations, I create a map which maps a pixel location to an index into the ordered object pixels. This allows me to easily and efficiently locate neighboring pixels in the Matlab ordering and in the linear system which uses the same ordering. The result is that there are only as many coefficients and as many variables in the linear system as there are pixels in the object. This can be a large savings (both in memory and computation) over incorporating every pixel in the image into the linear system.

Extensions

I did not implement any extensions.

Effort

I worked alone on this project. It took a fair amount of effort, but was manageable. I spent a significant amount of time trying to figure out why my system of equations for the reconstructed surface did not have a solution. Therefore, the project would have been easier and less frustrating had I not encountered that problem.

Significant Learning Outcomes

From carrying out this project, I have learned a few things worth mentioning. In general, I have now experienced a situation where bad data caused the program to fail rather than just corrupting the output. (Some of the normals in my normal map came out NaN, because normalizing a zero-length vector caused a divide-by-zero error. This fed bad values into the surface reconstruction system of equations, which then could not be solved.) This was significant because I do not have much experience with non-obvious cases of bad data causing program failure, especially when the bad data is generated by another part of my program. I will now be wiser and more likely to suspect this type of problem in the future.

Other general things I learned were more debugging approaches and how to better deal with large stumbling blocks. I was confident that I had the skills to be a successful graduate student. This assignment has shown me that, while that my be true, there is still much room for improvement.

I learned many things about programming with and using Matlab. I finally feel like I am becoming proficient enough with Matlab that I can use it for experiments and prototyping without it being a hindrance. I have expanded my tool set and increased my skills. Hopefully, this will benefit my work as a student and researcher. However, there is still much to learn.