Project 1: High Dynamic Range Imaging

Project 1: High Dynamic Range Imaging
Computer Vision – CS766 – Fall 2007

Brandon M. Smith

Outline

Introduction
Approach
Results
List of Submitted Files
Appendix A - image_list.txt
References

Introduction

Several algorithms were implemented for this project, including those for:

A. Aligning images automatically (bonus points)
B. Recovering the high dynamic range (HDR) radiance maps from digital photographs
C. Using the HDR radiance maps to create high dynamic range images

Two references were key to this project. The techniques outlined in Reference [1] were used for B and C above, and Reference [2] was used for A.

Approach

Image Alignment

It is often necessary to first align all of the images before finding the HDR radiance map. A program was written for doing this automatically based on [2]. It was written entirely in Java and is called Align. It is located in the alignment directory. To use the program, the user would type the following in UNIX:

java Align img1 img2 output

where img1 is the first image filename, img2 is the second image filename, and output is the image filename where the user would like the result stored. img2 is shifted the appropriate number of pixels to align it with img1. img2 is unchanged, however. Instead the new, shifted version of img2 is sent as a new image to output. The user can select img2 as output if she wishes, in which case img2 will be overwritten by a shifted version of itself.

HDR

Almost all of the implementation of the HDR technique described by Debevec [1] was written in Java. The small c++ portion is described below. To run the Java program, the user would type the following in UNIX:

java -Xms125m -Xmx1024m HDR img_list -s output

where

-Xms125m -Xmx1024m are necessary because, with large images like the ones in the images directory, an out-of-memory exception is thrown during execution otherwise. These arguments give the program adequate memory space to run.
HDR is the program.
img_list is a text file with the lambda value (described in [1]), the number of samples to use for recovering the HDR image map, and a list of all of the image files to use along with their shutter speed -- in that order. The file images/image_list.txt is this type of file and was used to create the submitted HDR image. It is shown in its entirety in Appendix A. To be more explicit, this text file has the following format:

<lambda>
<num samples>
<image1 filename> <1 / (image1 shutter speed)>
<image2 filename> <1 / (image2 shutter speed)>
...
<imageN filename> <1 / (imageN shutter speed)>

-s is an optional flag. When included, it tells the HDR command to print the output to a file. If omitted, the program will print all of the HDR image data to stdout.
output is a (temporary) output filename. The HDR image data will be sent here, but not in image form. The Java program HDR is not able to generate a .hdr file because there are no readily available Java libraries to do so. Note: including the filename here is optional. If it is omitted, the program will simply send output to a file called output.tmp, provided the -s is present (if the -s flag is not present, everything is sent to stdout).

The c++ portion of the HDR program is quite simple. It takes HDR image data via stdin and creates a properly formatted .hdr file using the gil library. To run the c++ program, the user would type the following in UNIX:

cat hdr_output | writeHDR output.hdr

where

cat hdr_output | sends the temporary output file generated by the Java program HDR to the writeHDR c++ program via stdin. hdr_output is the temporary output file itself and contains all of the HDR data.
writeHDR is the program.
output.hdr is the properly formatted .hdr file generated from the hdr_output file.

Combined, the two commands would look like this:

java -Xms125m -Xmx1024m HDR img_list.txt -s output.tmp
cat output.tmp | writeHDR result.hdr

A shorter way to write all this that avoids storing temporary data to a file, is to use the commands in conjunction like this:

java -Xms125m -Xmx1024m HDR img_list.txt | writeHDR result.hdr

However, the drawback to this is that the HDR Java program will not write any status updates to the console window and the user won't know how things are progressing. Since all of this is a bit cumbersome, a shell script was written to effectively create one combined command. The shell script, called simply hdr, looks like this:

#! /bin/csh -f

set filin = $argv[1] set fileout = $argv[2] java -Xms125m -Xmx1024m HDR $filin -s output.tmp cat output.tmp | writeHDR $fileout unset filein unset fileout rm output.tmp

To use this shell script the user would type the following in UNIX:

hdr images/images_list.txt images/result.hdr

The effect is that images/images_list.txt (see Appendix A) is used by the HDR program to open all of the images, which go through some computation to generate the HDR result, saved in images/result.hdr.

Results

Alignment

The results of the alignment program Align written for this assignment were mixed. On like images with similar shutter speeds it behaved well, getting the images within a couple pixels of perfect alignment. However, on images with relatively great differences in shutter speed, the program sometimes shifted the second image far too much -- 37 pixels too far in one case. It is possible that with further tweaking of the threshold value, which was set at -/+ 4 based on recommendations from [2], the results would have been better. This was of little concern, however, because the images were taken with great care in order to avoid movement. It was found that no alignment was needed to obtain good HDR results.

HDR

The HDR results obtained were quite good, as can be seen by comparing the untreated photographs with the tone-mapped version of the HDR recovered image below. The images shown are of the Wisconsin State Capitol building. In the left-most (dark) image we can see the individual stones making up the building wall, but we cannot see much detail on the light pole and railing in the foreground because they are too dark. On the other hand, the right-most (bright) image shows nice detail of the light pole and railing in the foreground, but the front of the building is now far too bright. A good compromise might be the middle image. However, in this image we still see that the front wall is too bright and the light pole and railing in front are a little too dark. The tone-mapped version of the HDR recovered image (shown larger and on the bottom), which was obtained from 27 images ranging in shutter speed from 1/2 second to 1/1000 second, solves this problem. We can see clearly both the dark areas (i.e., the light pole and railing in the foreground), and the bright areas (i.e., the front wall of the building). The tone mapping was done in HDR Shop using Reinhard's plugin [3].

shutter speed = 1/80 s

shutter speed = 1/25 s

shutter speed = 1/10 s

tone-mapped version of the HDR recovered image

A visual representation of the HDR radiance maps (one for each of red, green, and blue channels) are shown combined below. The data for these maps are generated automatically by the HDR Java program and are stored in a text file called gZij.txt. This text file was imported into Microsoft Excel -- the format of the file was designed to make this easy -- to create the plots. The vertical axis is the 0-255 pixel value and the horizontal axis is the log of the HDR radiance.

List of Submitted Files

alignment/ (directory)

Align.java

A Java program that aligns a second image with respect to the first image, then sends the output to a separate image. This syntax for using the Align command are shown in the Approach: Alignment section. The method outlined in [2] was used.

hdr* (shell script)

Hides the cumbersome nature of using the HDR Java program and the writeHDR c++ program in tandem. The syntax for using the script is shown in the Approach: HDR section.

Makefile

Used to compile everything in this directory. Typing make in UNIX within this directory will compile writeHDR.

include/ and libs/ (directories)

Contain the gil library, used for reading and writing images, including images in the .hdr format.

writeHDR*

The executable (command) that takes HDR data from stdin and sends it to a proper .hdr file.

writeHDR.cpp

The source code for the writeHDR command.

HDR.java

The source code that implements HDR.

Matrix.java

This file was taken from http://www.ee.ucl.ac.uk/~mflanaga/java/Matrix.html and was written by Michael Thomas Flanagan. I use it to create a matrix with which I can solve the system of equations.

flanagan/ (directory) and flanagan.jar

Contain a linear algebra package written by Michael Thomas Flanagan. This library is available at http://www.ee.ucl.ac.uk/~mflanaga/java I use it to solve the system of equations. Note: in order for Eclipse to build HDR.java, flanaga.jar must be included in the build path.

gZij.txt

This file is automatically generated by the HDR Java program. It contains the HDR radiance maps (one for each color component). The format of the file is easily discerned by looking at the file itself.

images/ (directory)

*.jpg

The image files, all the same picture taken at different shutter speeds.

image_list.txt

Used by the HDR Java program to get the lambda value, the number of samples to use, a list of all the image files to use, and their shutter speeds. See Appendix A.

result.hdr

The recovered HDR image, created from all of the *.jpg images by the HDR Java program and the writeHDR c++ program.

fav_artifact.jpg

This is the tone mapped image created from result.hdr using Reinhard's tone mapping plugin for HDR Shop [3]. All of the tone mapping values were left at default except for -key, which maps log average luminance. This was changed from 0.18 to 0.50.

gZij.gif

A plot of the HDR radiance map, g(Zij).

readme.txt

Explains how to run the HDR program.

Appendix A
images/image_list.txt

8 50 images/capitol/dsc01293.jpg 2 images/capitol/dsc01294.jpg 3 images/capitol/dsc01295.jpg 4 images/capitol/dsc01296.jpg 5 images/capitol/dsc01297.jpg 6 images/capitol/dsc01298.jpg 8 images/capitol/dsc01299.jpg 10 images/capitol/dsc01300.jpg 13 images/capitol/dsc01302.jpg 15 images/capitol/dsc01303.jpg 20 images/capitol/dsc01304.jpg 25 images/capitol/dsc01305.jpg 30 images/capitol/dsc01306.jpg 40 images/capitol/dsc01307.jpg 50 images/capitol/dsc01308.jpg 60 images/capitol/dsc01309.jpg 80 images/capitol/dsc01310.jpg 100 images/capitol/dsc01311.jpg 125 images/capitol/dsc01312.jpg 160 images/capitol/dsc01313.jpg 200 images/capitol/dsc01314.jpg 250 images/capitol/dsc01315.jpg 320 images/capitol/dsc01316.jpg 400 images/capitol/dsc01317.jpg 500 images/capitol/dsc01318.jpg 640 images/capitol/dsc01319.jpg 800 images/capitol/dsc01320.jpg 1000

References

[1] Debevec, Paul E. and Jitendra Malik. “Recovering High Dynamic Range Radiance Maps from Photographs.” Computer Science Division, University of California at Berkeley, Berkeley, CA 94720. Email: debevec@cs.berkeley.edu, malik@cs.berkeley.edu. More information and additional results may be found at: http://www.cs.berkeley.edu/~debevec/Research

[2] Ward, Greg. “Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Handheld Exposures.” Exponent – Failure Analysis Assoc., Menlo Park, CA.

[3] Reinhard HDR Tonemapping Plugin for HDR Shop, available at http://www.gregdowning.com/HDRI/tonemap/Reinhard/