-
diff --git a/assets/gradio/gradio_description_retargeting_video.md b/assets/gradio/gradio_description_retargeting_video.md
new file mode 100644
index 0000000..e54be9e
--- /dev/null
+++ b/assets/gradio/gradio_description_retargeting_video.md
@@ -0,0 +1,9 @@
+Retargeting
-Upload a Source Portrait as Retargeting Input, then drag the sliders and click the ๐ Retargeting button. You can try running it multiple times. +
Retargeting Image
+Upload a Source Portrait as Retargeting Input, then drag the sliders and click the ๐ Retargeting Image button. You can try running it multiple times.
๐ Set both target eyes-open and lip-open ratios to 0.8 to see what's going on!
+
+
diff --git a/assets/gradio/gradio_description_upload_animal.md b/assets/gradio/gradio_description_upload_animal.md
new file mode 100644
index 0000000..cefe812
--- /dev/null
+++ b/assets/gradio/gradio_description_upload_animal.md
@@ -0,0 +1,16 @@
+
+
+Retargeting Video
+Upload a Source Video as Retargeting Input, then drag the sliders and click the ๐ Retargeting Video button. You can try running it multiple times.
+
+ ๐ค Set target lip-open ratio to 0 to see what's going on!
+
+
diff --git a/inference.py b/inference.py
index 686dd81..5c80818 100644
--- a/inference.py
+++ b/inference.py
@@ -1,4 +1,7 @@
# coding: utf-8
+"""
+for human
+"""
import os
import os.path as osp
diff --git a/inference_animal.py b/inference_animal.py
new file mode 100644
index 0000000..8fddf7b
--- /dev/null
+++ b/inference_animal.py
@@ -0,0 +1,65 @@
+# coding: utf-8
+"""
+for animal
+"""
+
+import os
+import os.path as osp
+import tyro
+import subprocess
+from src.config.argument_config import ArgumentConfig
+from src.config.inference_config import InferenceConfig
+from src.config.crop_config import CropConfig
+from src.live_portrait_pipeline_animal import LivePortraitPipelineAnimal
+
+
+def partial_fields(target_class, kwargs):
+ return target_class(**{k: v for k, v in kwargs.items() if hasattr(target_class, k)})
+
+
+def fast_check_ffmpeg():
+ try:
+ subprocess.run(["ffmpeg", "-version"], capture_output=True, check=True)
+ return True
+ except:
+ return False
+
+
+def fast_check_args(args: ArgumentConfig):
+ if not osp.exists(args.source):
+ raise FileNotFoundError(f"source info not found: {args.source}")
+ if not osp.exists(args.driving):
+ raise FileNotFoundError(f"driving info not found: {args.driving}")
+
+
+def main():
+ # set tyro theme
+ tyro.extras.set_accent_color("bright_cyan")
+ args = tyro.cli(ArgumentConfig)
+
+ ffmpeg_dir = os.path.join(os.getcwd(), "ffmpeg")
+ if osp.exists(ffmpeg_dir):
+ os.environ["PATH"] += (os.pathsep + ffmpeg_dir)
+
+ if not fast_check_ffmpeg():
+ raise ImportError(
+ "FFmpeg is not installed. Please install FFmpeg (including ffmpeg and ffprobe) before running this script. https://ffmpeg.org/download.html"
+ )
+
+ fast_check_args(args)
+
+ # specify configs for inference
+ inference_cfg = partial_fields(InferenceConfig, args.__dict__)
+ crop_cfg = partial_fields(CropConfig, args.__dict__)
+
+ live_portrait_pipeline_animal = LivePortraitPipelineAnimal(
+ inference_cfg=inference_cfg,
+ crop_cfg=crop_cfg
+ )
+
+ # run
+ live_portrait_pipeline_animal.execute(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/readme.md b/readme.md
index 83606e5..61707ff 100644
--- a/readme.md
+++ b/readme.md
@@ -39,6 +39,7 @@
## ๐ฅ Updates
+- **`2024/08/02`**: ๐ธ We released a version of the **Animals model**, along with several other updates and improvements. Check out the details [**here**](./assets/docs/changelog/2024-08-02.md)!
- **`2024/07/25`**: ๐ฆ Windows users can now download the package from [HuggingFace](https://huggingface.co/cleardusk/LivePortrait-Windows/tree/main) or [BaiduYun](https://pan.baidu.com/s/1FWsWqKe0eNfXrwjEhhCqlw?pwd=86q2). Simply unzip and double-click `run_windows.bat` to enjoy!
- **`2024/07/24`**: ๐จ We support pose editing for source portraits in the Gradio interface. Weโve also lowered the default detection threshold to increase recall. [Have fun](assets/docs/changelog/2024-07-24.md)!
- **`2024/07/19`**: โจ We support ๐๏ธ **portrait video editing (aka v2v)**! More to see [here](assets/docs/changelog/2024-07-19.md).
@@ -55,7 +56,7 @@ This repo, named **LivePortrait**, contains the official PyTorch implementation
We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) ๐.
## Getting Started ๐
-### 1. Clone the code and prepare the environment
+### 1. Clone the code and prepare the environment ๐ ๏ธ
```bash
git clone https://github.com/KwaiVGI/LivePortrait
cd LivePortrait
@@ -64,56 +65,42 @@ cd LivePortrait
conda create -n LivePortrait python=3.9
conda activate LivePortrait
-# install dependencies with pip
# for Linux and Windows users
pip install -r requirements.txt
# for macOS with Apple Silicon users
pip install -r requirements_macOS.txt
```
-**Note:** make sure your system has [FFmpeg](https://ffmpeg.org/download.html) installed, including both `ffmpeg` and `ffprobe`!
+> [!Note]
+> Make sure your system has [`git`](https://git-scm.com/), [`conda`](https://anaconda.org/anaconda/conda), and [`FFmpeg`](https://ffmpeg.org/download.html) installed. For details on FFmpeg installation, see [**how to install FFmpeg**](assets/docs/how-to-install-ffmpeg.md).
-### 2. Download pretrained weights
+### 2. Download pretrained weights ๐ฅ
The easiest way to download the pretrained weights is from HuggingFace:
```bash
-# first, ensure git-lfs is installed, see: https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage
-git lfs install
-# clone and move the weights
-git clone https://huggingface.co/KwaiVGI/LivePortrait temp_pretrained_weights
-mv temp_pretrained_weights/* pretrained_weights/
-rm -rf temp_pretrained_weights
+# !pip install -U "huggingface_hub[cli]"
+huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights --exclude "*.git*" "README.md" "docs"
```
-Alternatively, you can download all pretrained weights from [Google Drive](https://drive.google.com/drive/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib) or [Baidu Yun](https://pan.baidu.com/s/1MGctWmNla_vZxDbEp2Dtzw?pwd=z5cn). Unzip and place them in `./pretrained_weights`.
-
-Ensuring the directory structure is as follows, or contains:
-```text
-pretrained_weights
-โโโ insightface
-โ โโโ models
-โ โโโ buffalo_l
-โ โโโ 2d106det.onnx
-โ โโโ det_10g.onnx
-โโโ liveportrait
- โโโ base_models
- โ โโโ appearance_feature_extractor.pth
- โ โโโ motion_extractor.pth
- โ โโโ spade_generator.pth
- โ โโโ warping_module.pth
- โโโ landmark.onnx
- โโโ retargeting_models
- โโโ stitching_retargeting_module.pth
+If you cannot access to Huggingface, you can use [hf-mirror](https://hf-mirror.com/) to download:
+```bash
+# !pip install -U "huggingface_hub[cli]"
+export HF_ENDPOINT=https://hf-mirror.com
+huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights --exclude "*.git*" "README.md" "docs"
```
+Alternatively, you can download all pretrained weights from [Google Drive](https://drive.google.com/drive/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib) or [Baidu Yun](https://pan.baidu.com/s/1MGctWmNla_vZxDbEp2Dtzw?pwd=z5cn) (WIP). Unzip and place them in `./pretrained_weights`.
+
+Ensuring the directory structure is as or contains [**this**](assets/docs/directory-structure.md).
+
### 3. Inference ๐
-#### Fast hands-on
+#### Fast hands-on (humans) ๐ค
```bash
-# For Linux and Windows
+# For Linux and Windows users
python inference.py
-# For macOS with Apple Silicon, Intel not supported, this maybe 20x slower than RTX 4090
+# For macOS users with Apple Silicon (Intel is not tested). NOTE: this maybe 20x slower than RTX 4090
PYTORCH_ENABLE_MPS_FALLBACK=1 python inference.py
```
@@ -136,12 +123,32 @@ python inference.py -s assets/examples/source/s13.mp4 -d assets/examples/driving
python inference.py -h
```
+#### Fast hands-on (animals) ๐ฑ๐ถ
+Animals mode is ONLY tested on Linux with NVIDIA GPU.
+
+You need to build an OP named `MultiScaleDeformableAttention` first, which is used by [X-Pose](https://github.com/IDEA-Research/X-Pose), a general keypoint detection framework.
+```bash
+cd src/utils/dependencies/XPose/models/UniPose/ops
+python setup.py build install
+cd - # equal to cd ../../../../../../../
+```
+
+Then
+```bash
+python inference_animal.py -s assets/examples/source/s39.jpg -d assets/examples/driving/wink.pkl --driving_multiplier 1.75 --no_flag_stitching
+```
+If the script runs successfully, you will get an output mp4 file named `animations/s39--wink_concat.mp4`.
+
+
+
+ Step 1: Upload a Source Animal Image (any aspect ratio) โฌ๏ธ
+
+
+
+
+ Step 2: Upload a Driving Pickle or Driving Video (any aspect ratio) โฌ๏ธ
+
+
+ Tips: Focus on the head, minimize shoulder movement, neutral expression in first frame.
+
+
+ 
+