Introduction
A sketch is a freehand drawing that is often used to graphically demonstrate an image, idea or principle. An experienced visual artist is able to draw a sketch rapidly; however, sketching a complex object or a detailed human face can be difficult and time-consuming for the untrained.
In this project, we study the relationship between a photo and a sketch of a human face (see Fig. 1). In particular, we are interested to automatically generate one from the other. Face sketch/photo synthesize can be used for face recognition, which has applications in law enforcement.
Fig. 1: Examples of face photos (left) and their corresponding sketches (right). The sketches were drawn by an artist.*
Face sketch synthesis is not easy. Artists often exaggerate distinctive facial features in a sketch and add shading to various facial features. It is therefore challenging to formulate rules for the automatic generation of a sketch and it might not be feasible to find a function that directly transforms a face photo to a face sketch.Methodology
To synthesize a face sketch, we use a database of sketch/photo images as examples (see Fig. 2). The input photo will first be divided into overlapping patches. For each of these patches, several candidate photo and sketch patch pairs are selected from the example dataset. The selection is based on the Euclidean distance of the photo patches. A new sketch patch is created using a linear combination of the selected candidate sketch patches. The key to this synthesis is the selection of the weights (w1 ... wK in Fig. 2) for the linear combination. To determine the weights, we minimize the sum of squared errors between each input photo patch and the linear combination of its candidate photo patches independently. The weights are then applied to the candidate sketch patches to form the final result. In order to obtain smooth adjacent sketch patches, we add an additional smoothness constrain for the overlapping part. The weights are determined by solving a Markov Weight Fields (MWF) model.
Fig. 2: Illustration of the proposed method. For a photo patch extracted from the input photo, we select K candidate photo sketch pairs from the training dataset (images in the middle). We then find the weights (w1 ... wK) to linear combine the selected candidate photo patches, such that the result is close to the input photo patch under the Euclidean distance. In addition, the overlapping parts of the adjacent synthesized sketch patches are constrained to be smooth. The estimated weights will then be applied to the candidate sketch patches and form the output sketch patch.
Due to its complexity, the MWF model cannot be solved using off-the-shelf methods. A novel Cascade Decomposition Method is proposed in which the original complex optimization is decomposed into several simplified optimization problems that can be solved in parallel.
The proposed method has the following contributions:
- New patches, which are not in the training dataset, can be synthesize by applying the linear combination of the candidate sketch patches. This is particular helpful when the training dataset is small.
- The MWF model is a convex quadratic programming problem. The optimal solution is easy to compute. In contrast, other methods employ the NP-hard Markov Random Fields model.
- The Cascade Decomposition Method solves the MWF model efficiently and significantly faster than comparable methods.
Results - Sketch Synthesize
Fig. 3 and Fig. 4 show various results obtained by the proposed method.
Fig. 3: Results obtained from frontal views.*
Fig. 4: Results obtained from side views. As the face photos in the training dataset are all in frontal view, some features of the face in the side view may not be found from the training dataset. Fortunately, a linear combination of selected candidate patches can synthesize new patches which are not in the training database.*
Results - Photo Synthesize
Using the proposed method, we can also synthesize a face photo given a face sketch. Fig. 5 shows the results of synthesizing a face photo using the proposed method.
Fig. 5: Results for synthesizing face photos using face sketch as input.*
Related project webpage: http://www.cs.hku.hk/facematchGrant Support: Matching Facial Composites to Partially Occluded or Disguised Faces for Law Enforcement and Crime Investigations (HK ITS/301/11)
For further information on this research topic, please contact Dr. Kenneth Wong.
*Input images are courtesy of CUFS. Reference: X. Wang and X. Tang, Face Photo-Sketch Synthesis and Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Vol. 31, 2009.