Project 3: Photometric Stereo
Jia Xu and Deepti Pachauri

Department of Computer Sciences, University of Wisconsin-Madison

1. Introduction

In this project, we implemented a system to construct a height field from a series of images of a diffuse object under different point light sources. Our software is able to calibrate the lighting directions, find the best fit normal and albedo at each pixel, then compute a surface which best matches the solved normals.


Our project is consist of four parts: calibrating light derections, normals from images, solving for color albedo and least square surface fitting. As for calibration, we use the chrome ball images to compute lighting directions. To infer the normals, we weight the solution by the pixel intensity as well as a weighting scheme, and use a least-squares technique to minimize the objective function. Then we use the normal to get albedo for each color channel. At last, we optimize the least square surface with the sparse matrices.

 

2. Algorithm


2.1 Calibrating


Our method of determining the direction of point light sources is to photograph a shiny chrome sphere in the same location as all the other objects. Firstly, we compute the centroid of the ball mask as the center of the ball and then compute the average distances between the boundary pixels to the center as the radius. After that we compute the centroid of the highlight pixels in each lighting image. Since we know the shape of this object, we can determine the normal at any given point on its surface, and therefore we can also compute the reflection direction for the brightest spot on the surface.


2.2. Normals from Images.


To help deal with shadows and noise in dark pixels, we firstly weight the solution by the pixel intensity. In order to get rid of the drawback of overweighting saturated pixels, we also add one weighting scheme. Then we get our objective function:

where the weighting scheme wi is:

The objective Q is then minimized with respect to g. Once we have the vector g = kd * n, the normalized direction gives n.


2.3. Solving for color albedo.


After obtaining a normal n for each pixel, we can solve for the albedos by another least squares solution. But this one ends up being a simple projection. The objective function is

To minimize it, differentiate with respect to kd, and set to zero:

Writing Ji = Li . n, we can also write this more concisely as a ratio of dot products: This can be done for each channel independently to obtain a per-channel albedo.


2.4. Least square surface fitting.


The goal of this part of the project is to best fit a surface given set of normals i.e., normals at each pixel. From the geometry of surfaces we know the following: given a tangent (edge) to a surface, the dot product of tangent and normal is zero. We exploited this property to solve for the surface in a least square sense. We used pixels belonging to mask (non-zero pixel values) for each image to compute constraint matrix M to solve for the Mz = v.

3. Implementation


3.1. Preprocessing.


For compatibility with matlab, we wrote a shell script convert.sh to convert all the .tga files to .png type. It used the sips command within Mac OS 10.6. To prepare the image data for photometric stereo procedures, we implemented one function [I, IGray, mask] = loadImages(imageSet, numOfImages), where imageSet is the folder containing numOfImages images. Here mask is a binary image whose original pixel value is above 200.


3.2. Calibrating Lighting Directions.


function L = getLightDirections() is used to get the light directions from the chrome sphere. Here we compute the average distances between the boundary pixels to the center as the radius, and then get the normal according to the difference between the centroid of the highlight pixels and the center of the ball. By initialized the view direction as R=[0;0;-1], we could get the light direction by L= -(2 * (N . R) * N - R).


3.3. Inferring Normals.


We tried to infer the normals by function N = getNormals( I,mask, L), which is based on the gray images. We solved the least square equation by computing A = wi * Ii .* L, b = wi * Ii * Ii. Then the normlized directions give normals.


3.4. Computing Albedo.


For each color channel, function A = getAlbedos(I, mask, L, N) computes albedo as a ratio of dot products: , where Ji = Li . n. Here, we also normlized the final albedos between 0 and 1.

3.5. Fitting Surface.


function z = getDepth(mask, N) did the least square surface fitting. Normal at each pixel provides two constraints (horizontal and vertical constraint) therefore for boundary pixel we decided to use one constraint only out of two. The system still remained overdetermined due to pixels which do not lie on the boundaries of the image.


3.6. Code Guideline.


We integrated all the functions in getPhotometricStereo.m . In this function, you will get the normals, color albedos and surfacc from imageSet based on L. To encode normals into RGB, we also performed the normlization. For plotting purpposes we used surfl command of matlab. To get a better view we also used imagesc using grey scale. You might started with test.m.

4. Results


4.1. RGB-encoded Normals.



4.2. Albedo Map.



4.3. Depth Map.



4.4. Different Views of the Reconstructed Surface: view([100 50]) and view([50 100])

5. Conclusion
  • When RGB-encoding the normals, we got a dark blue image after directly using the imshow function. As the normals contains negative value (for x and y direction), we normalized each direction into the interval [0,1]. Then we got a better output.
  • We also performed the normlization for albedo map. But you might also notice some hightlighted edges in the diffuse of buddha. We thought it might be helpful if we add a smoothing term.
  • While handling background pixel, we initially thought to assign 0 to all background pixel. MATLAB complained about rank deficiency. However when we assigned inf to all background pixels MATLAB did not complain.
    •  

    Reference