LivePortrait/assets/docs/changelog/2024-08-02.md

2024/08/02

Animals Singing Dance Monkey 🎤

🎉 We are excited to announce the release of a new version featuring animals mode, along with several other updates. Special thanks to the dedicated efforts of the LivePortrait team. 💪

Updates on Animals mode

We are pleased to announce the release of the animals mode, which is fine-tuned on approximately 230K frames of various animals (mostly cats and dogs). The trained weights have been updated in the liveportrait_animals subdirectory, available on HuggingFace or Google Drive. You should download the weights before running. There are two ways to run this mode.

Please note that we have not trained the stitching and retargeting modules for the animals model due to several technical issues. This may be addressed in future updates. Therefore, we recommend disabling stitching by setting the --no_flag_stitching option when running the model. Additionally, paste-back is also not recommended.

Before launching, ensure you have installed transformers==4.22.0, pillow>=10.2.0, which are already updated in requirements_base.txt. We have chosen XPose as the keypoints detector for animals. This relies on transformers and requires building an OP named MultiScaleDeformableAttention by

cd src/utils/dependencies/XPose/models/UniPose/ops
python setup.py build install
cd - # equal to cd ../../../../../../../

You can run the model using the script inference_animals.py:

python inference_animals.py -s assets/examples/source/s39.jpg -d assets/examples/driving/wink.pkl --no_flag_stitching --driving_multiplier 1.75

Alternatively, we recommend using Gradio. Simply launch it by running:

python app_animals.py # --server_port 8889 --server_name "0.0.0.0" --share

Warning

XPose is only for Non-commercial Scientific Research Purposes, you should remove and replace it with other detectors if you use it for commercial purposes.

Updates on Humans mode

• Driving Options: We have introduced an expression-friendly driving option to reduce head wobbling, now set as the default. While it may be less effective with large head poses, you can also select the pose-friendly option, which is the same as the previous version. This can be set using --driving_option or selected in the Gradio interface. Additionally, we added a --driving_multiplier option to adjust driving intensity, with a default value of 1, which can also be set in the Gradio interface.

• Retargeting Video in Gradio: We have implemented a video retargeting feature. You can specify a target lip-open ratio to adjust the mouth movement in the source video. For instance, setting it to 0 will close the mouth in the source video 🤐.

Others

• Poe supports LivePortrait. Check out the news on X.
• ComfyUI-LivePortraitKJ (1.1K 🌟) now includes MediaPipe as an alternative to InsightFace, ensuring the license remains under MIT and Apache 2.0.

Below are some screenshots of the new features and improvements:

The Gradio Interface of Animals Mode

Driving Options and Multiplier

The Feature of Retargeting Video