Rig-Your-Portrait: Controllable and Relightable Portrait Video Generation with Explicit 3D Guidance

🚗 Model Architecture and Parameter Initialization

The model network comprises a VAE (including encoder and decoder, i.e., VAE enc. and VAE dec.), a CLIP image encoder, 3D face reconstruction models, DenoisingNet, ReferenceNet, an expression guidance encoder (Exp enc.), a geometry guidance encoder (Geo enc.), a texture guidance encoder (Tex enc.), and a Lambertian illumination guidance encoder (Lamb enc.).

The architecture and weights of the VAE, DenoisingNet, and ReferenceNet are inherited from Stable Diffusion, while the temporal layer of DenoisingNet is initialized with the pretrained weights from AnimateDiff. The 3D face reconstruction models are SMIRK and DECA, used during the data processing stage. Additionally, the expression, geometry, texture, and illumination guidance encoders share the same lightweight architecture, utilizing four convolutional layers for feature extraction.

🌲 Training Stage

The model is trained on a small subset of publicly available datasets, including a limited number of portrait videos with different identities, expressions, and head poses, as well as partial indoor multi-illumination images from Multi-PIE. Multi-PIE, which consists of multi-view static illumination portraits captured under indoor white light with limited incident angles, is used to preliminarily validate the feasibility of illumination control. The model's loss function inherits from Animate Anyone, with the training process being conducted in two stages.

In the first training stage, DenoisingNet (excluding the temporal layer), ReferenceNet, and the expression, geometry, texture, and illumination guidance encoders are optimized. The reference image \( I_s \) is randomly selected from the entire portrait video clip or pseudo-video clip (where Multi-PIE images of the same identity and camera pose under varying illumination conditions are treated as a video).

3D explicit guidance maps are generated by extracting FLAME parameters from the video frame \( I_d \) using 3D face reconstruction models:

• Expression map \( cond_{exp} \) is extracted and rendered using SMIRK and its renderer.
• Geometry map \( cond_{geo} \), texture map \( cond_{tex} \), and Lambertian illumination map \( cond_{lamb} \) are extracted and rendered using DECA and its renderer.

Model inputs include \( I_s \) and the rendered guidance sequences (expression, geometry, texture, illumination) from \( I_d \).

The 3D explicit guidance maps are processed by their respective encoders (Exp enc., Geo enc., Tex enc., Lamb enc.). The extracted features are summed, then fused with the noisy latent 'Noise', and injected into DenoisingNet.

The model extracts appearance features from the reference image using ReferenceNet, integrating them into DenoisingNet via spatial-attention. Simultaneously, the CLIP image encoder encodes the semantic features of the reference image for cross-attention.

In the second training stage, the temporal layer is incorporated after the spatial-attention and cross-attention components in DenoisingNet. Only the temporal layer is updated, while all other parameters remain frozen.

✨ Inference Stage

During inference, all network weights are frozen. For facial attribute transfer, FLAME parameters are extracted from both the reference image \( I_s \) and driving sequence \( I_d \). When specifying attributes (e.g., expression and head pose), the reference portrait's expression and pose coefficients are replaced with those from the driving sequence. The modified FLAME parameters are rendered into 3D guidance maps, which are sequentially fed into the guidance encoders (expression, geometry, texture, illumination) as driving signals. Specifically, by keeping the identity parameters in the FLAME parameters of the reference portrait unchanged, the animated image can better preserve the facial shape of the reference portrait, thereby alleviating the identity leakage issue.

The first-stage model enables controllable and relightable portrait editing by transferring specified attributes (expression, head pose, or illumination) from the target image to the reference portrait. The second-stage model extends this to portrait video generation by transferring the facial attributes from the driving video to the reference portrait.

🖼️ The first-stage model could disentangledly transfer the facial attributes, such as expressions, head poses, and illumination conditions, from the target portrait to the reference portrait.

As shown above, we use the first-stage model to perform facial attribute transfer on test images from the FFHQ dataset, which was not involved in training. The results show that the model can transfer the target portrait's expression, pose, and illumination attributes to the reference portrait—individually or simultaneously—while preserving the reference portrait's identity features such as facial shape and background.

🎞️ Qualitative results of self-reenactment using the second-stage model.

🎞️ Qualitative results of cross-reenactment using the second-stage model.

As shown above, self-reenactment and cross-reenactment experiments demonstrate simultaneous expression and head pose transfer using the second-stage model. The reference images and the driving videos are from the TalkingHead-1KH and VFHQ test sets. Preliminary qualitative results indicate the method's feasibility and potential for improvement on larger training sets.

🖼️ The inference results of the first-stage model when using the original 3D explicit maps of another test portrait as driving signals.

As demonstrated above, we also present the inference results of the first-stage model (trained without Multi-PIE) when using the original 3D explicit maps of another test portrait as driving signals. As anticipated, the facial shape in the generated results differs from that of the reference images.

        @misc{zhang2025rigyourportrait,
            title={Rig-Your-Portrait: Controllable and Relightable Portrait Video Generation with Explicit 3D Guidance},
            author={Dingyun Zhang},
            year={2025},
            howpublished={\url{https://rigyourportrait.github.io}},
        }