Chris Hinrichs
CS 766 Project 1 report.

Compiling and running requirements

I used the LAPACK++ library for C++ to solve the linear system of equations in my project. The required library is lapackpp.so, and can be found on the web page linked off of the project description page. For image importation, exportation, and gray-level calculation I used libgil, also provided through the project description web page. Various other libraries are required, and I have placed them all in ~hinrichs/lib.

The program uses a certain format of file to tell it which pictures to use, and the exposure time for each. The first line contains a prefix of the locations of all of the files. The next line contains an extension, (e.g. png or jpg) which is appended to each file name. The third line contains a number which says how many more lines to read, i.e. how many pictures to use. After than, every line consists of 1/[exposure time in seconds] followed by a space and the number of the file in the sequence. The file name for each picture is constructed by concatenating the prefix, then the number, then a "." then the extension.

Algorithm

I used the algorithm in [Debevec '97] to fit the photometric response curve for each series of pictures, and use it to interpolate back to the scene irradiance values for each pixel. The algorithm takes P pictures with known exposure lengths, chooses N pixels, (or they can be hand chosen for it,) and tries to induce the actual exposure values as well as the (log of the) function of pixel values to scene irradiance, g(z), (which is the inverse of the function of irradiance to pixel values.) Since there are only a finite number of values in the domain of this function, it is relatively easy to learn, rather than its inverse. The N*P pixels are then used in conjunction with the exposure times to make g(z) closely approximate log(E) + log(t) where E is the amount of light coming in at a given pixel, and t is the amount of time the picture was exposed to the light. Even though g(z) is a non-linear function it can be induced by fitting a system of linear equations because we only induce it over a finite number of values, so it can be thought of as a piece-wise linear approximation of g(z). The variables in this system are the values of g(z) and E for each pixel. The constraints are the requirements that each of the N*P pixels must be close to the chosen curve, and that each of the values of z(g) must be close to each other.

Once g(z) has been induced, re interpolating the scene irradiance values is done by averaging exp(g(z)) over all pictures at the same pixel. This gives a high-dynamic range radiance map of the scene, and other things can be done with it.

In order to make the system more robust, I experimented with automatically choosing pixels. Originally, I tried to choose pixels from regions with low variance, which is one of the recomendations in [Debevec '97], however I was too strict about enforcing low variance and only got pixels from a subset of the range of the scene. I then relaxed the constraint so that the pixels were essentially chosen at random, which improved the performance of the algorithm dramatically, but it was still necessary to manually choose points from the regions with the highest dynamic range so that in the HDR image those regions could be reasonably represented. I then implemented a scheme which breaks the range into buckets, divides the range evenly between buckets, and chooses the same number of pixels in each bucket by averaging the pixel's value over all pictures. This has the drawback that if the entire range of 8-bit values is not represented in the pictures, then a bucket might not have enough pixels in the entire picture, which would create an infinite loop as the algorithm tries to randomly choose pixels until it finds enough to fill the bucket. This may not be such a disaster, however, as most sets of pictures will have at least some bright and dark pixels everywhere. An improvement on this could be to put more pixels in the buckets near the middle, rather than evenly apportioning them.

Things learned

In implementing this algorithm I gained experience with the Singular Value Decomposition method of solving an overdetermined system of equations in the least-squares sense. I also gained the insight that non-linear functions can very usefully be approximated by linear piece-wise systems.

Probably the most surprising thing, (except in hindsight) is that the choice of the N pixels has a strong effect on the quality of results produced. It turns out that if the pixels chosen do not represent the entire range, then the function can be highly skewed. In pathological cases, e.g. where all pixels are chosen from a small part of the range, the function is not even close to being monotonic.

Things left unexplained

My algorithm seems to generate very unreasonable values for E, despite generating reasonable values for g(z). Usually, there will be one value of E which it reports as 10^300 and the rest round down to 0.

Results

One artifact generated by my algorithm appears when there is insufficient data about detail in high-intensity areas, which can happen if not enough low-intensity photographs are available. The artifact which appears is a kind of diagonal striping in brightly lit areas. One example can be seen here:

The artifact is probably some kind of residual information created when high-intensity regions are clamped down to the maximum pixel value, so detailed information must come from low-exposure images, where even bright areas are mapped to medium pixel values. If low exposure images are not given to the algorithm, then the artifact becomes much more pronounced.

Otherwise, the HDR maps produced for test data seem to be reasonable.

References