This is the code repo of facial details synthesis from single input image. Paper here, Supplemental Material: here.

This repository consists 5 individual parts: DFDN, emotionNet, landmarkDetector, proxyEstimator and faceRender. The DFDN is based on junyanz's pix2pix, for the landmark and expression detector, we use a simplify version of openFace, and our proxyEstimator is modified based on patrikhuber's fantastic work eos . We want to thank each of them for their kindly work.

Facial Details Synthesis

Anpei Chen*, Zhang Chen*, Guli Zhang, Ziheng Zhang, Kenny Mitchell, Jingyi Yu

We present a single-image 3D face synthesis technique that can handle challenging facial expressions while recovering fine geometric details. Our technique employs expression analysis for proxy face geometry generation and combines supervised and unsupervised learning for facial detail synthesis. On proxy generation, we conduct emotion prediction to determine a new expression-informed proxy. On detail synthesis, we present a Deep Facial Detail Net (DFDN) based on Conditional Generative Adversarial Net (CGAN) that employs both geometry and appearance loss functions. For geometry, we capture 366 high-quality 3D scans from 122 different subjects under 3 facial expressions. For appearance, we use additional 163K in-the-wild face images and apply image-based rendering to accommodate lighting variations. Comprehensive experiments demonstrate that our framework can produce high-quality 3D faces with realistic details under challenging facial expressions.

Features

• Functionality
  • proxy estimation with expression/emotion prior
  • facial details prediction, i.e. winkles
  • results visualizer or facial render
• Input: single image or images folder
• Output: proxy mesh & texture, details displacementMap and normalMap
• OS: Window 10

Set up environment

1. We specify python3 and pytorch

   Or you can use Ananconda to create new environment in root directory by bash conda create -n facial_details --file requirement.txt

Released version

1. Download released package.

2. Download models and pre-train weights. DFDN checkpoints, unzip to ./DFDN/checkpoints landmork models, unzip to ./landmarkDetector [Optional] emotionNet checkpoints, unzip to ./emotionNet/checkpoints

3. Install BFM2017

   • install eos by pip install --force-reinstall eos-py==0.16.1

   • Download BFM2017 and copy model2017-1_bfm_nomouth.h5 to ./proxyEstimator/bfm2017/.

   • Run python convert-bfm2017-to-eos.py to generate bfm2017-1_bfm_nomouth.bin in bfm2017 folder.

4. Have fun!

Usage

• For proxy estimation,

python proxyPredictor.py -i path/to/input/image -o path/to/output/folder [--FAC 1][--emotion 1]

• For batch processing, you can set -i to a image folder.

• For prior features, you can optional choose one of those two priors: with facial coding features, type --FAC 1, with emotion features type --emotion 1.

example: python proxyPredictor.py -i ./samples/proxy -o ./results

• For facial details estimation,

python facialDetails.py -i path/to/input/image -o path/to/output/folder

example:

python facialDetails.py -i ./samples/details/019615.jpg -o ./results

python facialDetails.py -i ./samples/details -o ./results

Compiling

we suggest you directly download the released package for convenient, if you are interested in compile the source code, please follow the following guidelines:

on the way .....

Citation

If you find this code useful to your research, please consider citing:

@InProceedings{Chen_2019_ICCV,  
author = {Chen, Anpei and Chen, Zhang and Zhang, Guli and Mitchell, Kenny and Yu, Jingyi},  
title = {Photo-Realistic Facial Details Synthesis From Single Image},  
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},  
month = {Oct},  
year = {2019}  
}