Denoising versus Deblurring:

Introduction

Compared to early digital cameras which have only 8-bit sensors, latest digital cameras such as Canon 7D and Pointgrey Grasshopper have 14-bit sensors; many others have 12-bit sensors. These new sensors offer new oppotunities for flexible HDR imaging using a handheld moving camera. For these sensors with high-precesion quantization, the key challenge in HDR imaging is to obtain reliable less significant bits for dark pixels because they encode much random noise. We consider the following two processes for this purpose:

1. Capturing a single image with a long exposure time T and a low-ISO setting. This image has low noise and high dynamic range (assuming the camera has a 14-bit sensor), but often contains motion blur. We need to remove the blur to obtain the underlying HDR image.
2. Capturing a sequence of images in a burst mode within the same period T. Each image has a short exposure time and a high-ISO setting; consequently, each image is sharp but noisy. We need to remove the noise to recover the HDR images.

We present a novel theoretical analysis which indicates that multi-frame denoising is in general a more reliable approach than deblurring to HDR imaging for the following two reasons:

1. Multi-frame denoising is a numerically much better conditioned problem than deconvolution within a large range of camera overhead (from 0 to 90%). Camera overhead is the time required to transfer one image from sensor to storage between two consecutive image capture.
2. Denoising multiple high-ISO images reduces read noise more effectively than a single-shot low-ISO imaging, if post-amplifier read noise is not trivially small compared to the pre-amplifier read noise, which is true for popular high-end cameras like Canon 5D and 1D Mark III.

Our analysis also suggests, conventional HDR approach that fuses shots with varying exposures is less reliable than multi-frame denoising, because shots with long exposure need to be deblurred. Our theoretical analysis assumes translational motion and spatially-invariant blur.

To put our analytical insight to practical use, we demonstrate that existing optical flow and image denoising techniques can be combined for HDR imaging, which enables capturing HDR images for complex scenes with large depth variation using handheld cameras. Such scenes cause spatially-varying motion blur for moving cameras, which poses a serious challenge to existing HDR techniques. Our HDR images are sharp and clean and the HDR videos have high temporal stability, as the following videos show. Our technique currently is limited to handshake motion and small scene motion; handling highly dynamic scenes, such as a dancing crowd, remains as a future work.

Publication

Li Zhang, Alok Deshpande, Xin Chen, Denoising versus Deblurring: HDR Techniques Using Moving Cameras, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June, 2010. [PDF (8MB), A note that reconciles our conclusions with Hasinoff et al 2009.]

Acknowledgement

This work is supported in part by National Science Foundation EFRI-0937847, IIS-0845916, and IIS-0916441.

Demonstration Videos (Click on the video to play it)

The following two videos show input noisy videos on the left and output tone-maped HDR videos on the right. The input videos are captured by a Pointgrey Grasshopper 14S3D color video camera with F8 aperture, short exposture time (0.56 milisecond) and highest gain setting. (To visulize the amount of noise in the input, follow the links below in the "HDR Image Results" section.)

Download the MOV file (8.5MB).

Download the MOV file (14.8MB).

HDR Image Results on Real Scenes

The following examples give a more in-depth illustration of our HDR method, in terms of the amount noise in the input, output PSNR, comparision with the state-of-the-art, and the importance of 14-bit sensors for HDR imaging.

HDR Imaging by Multi-Image Denoising
These examples show that HDR images can be generated by denoising multiple moving images captured by a camera with high-resoluton Analog-to-Digital conversion.

Books Results

Model Results

Cake Results

Our Multi-Image Denoising vs. VBM3D
These examples show that our approach is more effective than the state-of-the-art video denoising algorithm for the purpose of generating HDR images captured by a hand-held moving camera.

Books Results

Model Results

Cake Results

The Effect of 8-bit vs. 14-bit Analog-to-Digital Conversion on Our Results
These results show that having a high-resolution ADC is important for computing HDR images by multi-image denoising.

Model Results

Cake Results

Simulation Experiments to Verify Our Theory

Please read our paper for details of our theoretical analysis.

Single-Image Deblurring vs. Multi-Image Denoising
These examples show that multi-image denoising is a much more reliable way for image restoration than single-image deblurring.

Blackboard Results, Flower Results, Garden Gate Results, Kid Sweater Results

The Effect of Camera Overhead on Multi-Image Denoising
These examples show how multi-image denoising performace varies as a function of camera overhead -- the time needed to save images from a sensor to its storage.

Blackboard Results, Flower Results, Garden Gate Results, Kid Sweater Results