4.2 KiB
2024/08/02
Animals Singing Dance Monkey 🎤 |
🎉 We are excited to announce the release of a new version featuring animals mode, along with several other updates. Special thanks to the dedicated efforts of the LivePortrait team. 💪
Updates on Animals mode
We are pleased to announce the release of the animals mode, which is fine-tuned on approximately 230K frames of various animals (mostly cats and dogs). The trained weights have been updated in the liveportrait_animals
subdirectory, available on HuggingFace or Google Drive. You should download the weights before running. There are two ways to run this mode.
Please note that we have not trained the stitching and retargeting modules for the animals model due to several technical issues. This may be addressed in future updates. Therefore, we recommend disabling stitching by setting the
--no_flag_stitching
option when running the model. Additionally,paste-back
is also not recommended.
Before launching, ensure you have installed transformers==4.22.0
, pillow>=10.2.0
, which are already updated in requirements_base.txt
. We have chosen XPose as the keypoints detector for animals. This relies on transformers
and requires building an OP named MultiScaleDeformableAttention
by
cd src/utils/dependencies/XPose/models/UniPose/ops
python setup.py build install
cd - # equal to cd ../../../../../../../
You can run the model using the script inference_animals.py
:
python inference_animals.py -s assets/examples/source/s39.jpg -d assets/examples/driving/wink.pkl --no_flag_stitching --driving_multiplier 1.75
Alternatively, we recommend using Gradio. Simply launch it by running:
python app_animals.py # --server_port 8889 --server_name "0.0.0.0" --share
Warning
XPose is only for Non-commercial Scientific Research Purposes, you should remove and replace it with other detectors if you use it for commercial purposes.
Updates on Humans Mode
-
Driving Options: We have introduced an
expression-friendly
driving option to reduce head wobbling, now set as the default. While it may be less effective with large head poses, you can also select thepose-friendly
option, which is the same as the previous version. This can be set using--driving_option
or selected in the Gradio interface. Additionally, we added a--driving_multiplier
option to adjust driving intensity, with a default value of 1, which can also be set in the Gradio interface. -
Retargeting Video in Gradio: We have implemented a video retargeting feature. You can specify a
target lip-open ratio
to adjust the mouth movement in the source video. For instance, setting it to 0 will close the mouth in the source video 🤐.
Others
- Poe supports LivePortrait. Check out the news on X.
- ComfyUI-LivePortraitKJ (1.1K 🌟) now includes MediaPipe as an alternative to InsightFace, ensuring the license remains under MIT and Apache 2.0.
Below are some screenshots of the new features and improvements:
The Gradio Interface of Animals Mode |
Driving Options and Multiplier |
The Feature of Retargeting Video |