Face Image Retrieval by Shape Manipulation

Brandon M. Smith Shengqi Zhu Li Zhang

University of Wisconsin, Madison



Current face image retrieval methods achieve impressive results, but lack efficient ways to refine the search, particularly for geometric face attributes. Users cannot easily find faces with slightly more furrowed brows or specific leftward pose shifts, for example. To address this problem, we propose a new face search technique based on shape manipulation that is complementary to current search engines. Users drag one or a small number of contour points, like the bottom of the chin or the corner of an eyebrow, to search for faces similar in shape to the current face, but with updated geometric attributes specific to their edits. For example, the user can drag a mouth corner to find faces with wider smiles, or the tip of the nose to find faces with a specific pose. As part of our system, we propose (1) a novel confidence score for face alignment results that automatically constructs a contour-aligned face database with reasonable alignment accuracy, (2) a simple and straightforward extension of PCA with missing data to tensor analysis, and (3) a new regularized tensor model to compute shape feature vectors for each aligned face, all built upon previous work. To the best of our knowledge, our system demonstrates the first face retrieval approach based chiefly on shape manipulation. We show compelling results on a sizeable database of over 10,000 face images captured in uncontrolled environments.

Brandon M. Smith, Shengqi Zhu, Li Zhang. Face Image Retrieval by Shape Manipulation, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June, 2011. [PDF 1.1MB]
Related Publication
Yiqing Yang, Li Zhang, Sen Wang, Hongrui Jiang, Chris J. Murphy, Jim Ver Hoeve. A Multi-Affine Model for Tensor Decomposition, IEEE workshop on Information Theory in Computer Vision and Pattern Recognition, 2011. [PDF 400KB ]
This work is supported in part by NSF IIS-0845916, NSF IIS-0916441, a Sloan Research Fellowship, a Packard Fellowship for Science and Engineering, and a gift donation from Eastman Kodak Company. Brandon Smith is also supported by an NSF graduate fellowship.
Supplemental Video
Download [MP4 19 MB]
Face Retrieval Video
iPad Demo Video
Download [MP4 21 MB]
iPad Demo Video