mirror of
https://github.com/KwaiVGI/LivePortrait.git
synced 2024-12-23 13:04:23 +00:00
76 lines
4.7 KiB
Markdown
76 lines
4.7 KiB
Markdown
## 2024/08/02
|
|
|
|
<table class="center" style="width: 80%; margin-left: auto; margin-right: auto;">
|
|
<tr>
|
|
<td style="text-align: center"><b>Animals Singing Dance Monkey 🎤</b></td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td style="border: none; text-align: center;">
|
|
<video controls loop src="https://github.com/user-attachments/assets/38d5b6e5-d29b-458d-9f2c-4dd52546cb41" muted="false" style="width: 60%;"></video>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
|
|
|
|
🎉 We are excited to announce the release of a new version featuring animals mode, along with several other updates. Special thanks to the dedicated efforts of the LivePortrait team. 💪 We also provided an one-click installer for Windows users, checkout the details [here](./2024-08-05.md).
|
|
|
|
### Updates on Animals mode
|
|
We are pleased to announce the release of the animals mode, which is fine-tuned on approximately 230K frames of various animals (mostly cats and dogs). The trained weights have been updated in the `liveportrait_animals` subdirectory, available on [HuggingFace](https://huggingface.co/KwaiVGI/LivePortrait/tree/main/) or [Google Drive](https://drive.google.com/drive/u/0/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib). You should [download the weights](https://github.com/KwaiVGI/LivePortrait?tab=readme-ov-file#2-download-pretrained-weights) before running. There are two ways to run this mode.
|
|
|
|
> Please note that we have not trained the stitching and retargeting modules for the animals model due to several technical issues. _This may be addressed in future updates._ Therefore, we recommend **disabling stitching by setting the `--no_flag_stitching`** option when running the model. Additionally, `paste-back` is also not recommended.
|
|
|
|
#### Install X-Pose
|
|
We have chosen [X-Pose](https://github.com/IDEA-Research/X-Pose) as the keypoints detector for animals. This relies on `transformers==4.22.0` and `pillow>=10.2.0` (which are already updated in `requirements.txt`) and requires building an OP named `MultiScaleDeformableAttention`.
|
|
|
|
Refer to the [PyTorch installation](https://github.com/KwaiVGI/LivePortrait?tab=readme-ov-file#for-linux-or-windows-users) for Linux and Windows users.
|
|
|
|
|
|
Next, build the OP `MultiScaleDeformableAttention` by running:
|
|
```bash
|
|
cd src/utils/dependencies/XPose/models/UniPose/ops
|
|
python setup.py build install
|
|
cd - # this returns to the previous directory
|
|
```
|
|
|
|
To run the model, use the `inference_animals.py` script:
|
|
```bash
|
|
python inference_animals.py -s assets/examples/source/s39.jpg -d assets/examples/driving/wink.pkl --no_flag_stitching --driving_multiplier 1.75
|
|
```
|
|
|
|
Alternatively, you can use Gradio for a more user-friendly interface. Launch it with:
|
|
```bash
|
|
python app_animals.py # --server_port 8889 --server_name "0.0.0.0" --share
|
|
```
|
|
|
|
> [!WARNING]
|
|
> [X-Pose](https://github.com/IDEA-Research/X-Pose) is only for Non-commercial Scientific Research Purposes, you should remove and replace it with other detectors if you use it for commercial purposes.
|
|
|
|
### Updates on Humans mode
|
|
|
|
- **Driving Options**: We have introduced an `expression-friendly` driving option to **reduce head wobbling**, now set as the default. While it may be less effective with large head poses, you can also select the `pose-friendly` option, which is the same as the previous version. This can be set using `--driving_option` or selected in the Gradio interface. Additionally, we added a `--driving_multiplier` option to adjust driving intensity, with a default value of 1, which can also be set in the Gradio interface.
|
|
|
|
- **Retargeting Video in Gradio**: We have implemented a video retargeting feature. You can specify a `target lip-open ratio` to adjust the mouth movement in the source video. For instance, setting it to 0 will close the mouth in the source video 🤐.
|
|
|
|
### Others
|
|
|
|
- [**Poe supports LivePortrait**](https://poe.com/LivePortrait). Check out the news on [X](https://x.com/poe_platform/status/1816136105781256260).
|
|
- [ComfyUI-LivePortraitKJ](https://github.com/kijai/ComfyUI-LivePortraitKJ) (1.1K 🌟) now includes MediaPipe as an alternative to InsightFace, ensuring the license remains under MIT and Apache 2.0.
|
|
- [ComfyUI-AdvancedLivePortrait](https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait) features real-time portrait pose/expression editing and animation, and is registered with ComfyUI-Manager.
|
|
|
|
|
|
|
|
**Below are some screenshots of the new features and improvements:**
|
|
|
|
|  |
|
|
|:---:|
|
|
| **The Gradio Interface of Animals Mode** |
|
|
|
|
|  |
|
|
|:---:|
|
|
| **Driving Options and Multiplier** |
|
|
|
|
|  |
|
|
|:---:|
|
|
| **The Feature of Retargeting Video** |
|