DETAILED FACIAL PORTAITS USING TEXTURE TRANSFER

Kenneth Jones

CS 638

5/5/10

Motivation

• Facial portraits in the Image Quilting paper by Efros looked promising if could capture more details
• WIRED magazine artistic author portraits
• Can texture transfer be extended for this particular task?

Texture Synthesis Algorithm

• Go through image to be synthesized in raster scan order in steps of one block size (minus overlap)
• For every location, search the input texture for a set of blocks that satisfy overlap constraints within some error tolerance
• Find minimum cost path along the SSD error surface in this overlap
• Paste block onto texture.

Texture Transfer Extension

• Create correspondence map
  • Spatial map of some corresponding quantity over both the texture source and controlling target image
• When calculating overlap error, add this constraint
• Efros used the image intensities (luminance)

Use Empirical Approach

• Don’t overly bias texture selection
  • Started with 260 various potential textures
  • Ran multiple tile sizes for each
  • Narrowed down to 20 textures based on results
• Brute force enumeration on selected textures
  • Generated ~3000 parameter combinations with 4 output transformations for each = 12000+ images (~2.5 GB)
  • Visual inspection of results
• Task is trivial to parallelize, run one Matlab instance per core on subset of textures (still took multiple days of computation with 4 cores)

Sample of Textures

• Homogeneous

Sample of Textures

• Heterogeneous

Extension Number #1

• What is a pixel in a texture versus a controlling target image?
  • SSD error using pixel to pixel comparison too variable
• Using multiple texture pixels to match single controlling pixel provides better results
• Resize controlling image to higher resolution
• Found scale by 3x worked well

Image Resizing Results (2x)

Image Resizing Results (3x)

Image Resizing Results (4x)

Extension Number #2

• Discrete image intensities cause bumpy error surface
• Use Gaussian filter to blur both texture and target intensities to smooth error function

Blurred Intensity Results

Blurred Intensity Results

Blurred Intensity Results

Extension Number #3

• Raw image intensities don’t always match well between texture and target
  • Skin versus hair typically exhibits large contrast
  • Want to encourage heterogeneous element separation in texture source image.
• Scale source texture to match target
  • Perform histogram equalization by minimizing T according to |c₁(T(k))-c₀(k)| ,where c₀ is the cumulative histogram of the texture and c₁ is the cumulative sum of target histogram for all intensities k

Scaled Intensity Results

Scaled Intensity Results

Scaled Intensity Results

Scaled Intensity Results

Extension Number #4

• Skin typically has multiple shades in portrait
• Pre-process texture image by artificially shading segment to encourage better match
  • Want to keep enough untouched texture synthesis so only use 25% of area
  • Used linear decrease in intensity (just one of many options)
  • Too much shading looks fake so only decrease by max of ~20% (variable depending on texture)

Shading Results

Shading Results

Shading Results

Extension Number #5

• Try another approach to match texture palette to target
• Post-process synthesized image to blend different amount for each RGB channel in target based on mean of each channel in texture
• This modified alpha blending keeps target from bleeding in too much

Channel Blending Results

Channel Blending Results

Channel Blending Results

Channel Blending Results

Channel Blending Results

Channel Blending Results

Channel Blending Results

Extension Number #6

• Create artistic portrait by using black and white synthesized result as canvas
• Post-process normal output to grayscale
• Alpha blend with original (15%)

B&W blending Results

B&W blending Results

B&W blending Results

B&W blending Results

Questions