Bring portraits to life!
Go to file
Maki 8876f16ea5 🐳 [feat] Enhance Docker setup for LivePortrait
- Updated the base image to NVIDIA CUDA 12.0.1 and installed essential system packages.
- Upgraded pip and installed required Python libraries from requirements.txt.
- Set the Asia/Tokyo timezone and specified the working directory to /LivePortrait.
2024-07-12 16:19:03 +09:00
.vscode feat: launch LivePortrait 2024-07-04 04:32:47 +08:00
assets doc: update readme and changelog 2024-07-12 14:45:24 +08:00
docker 🐳 [feat] Enhance Docker setup for LivePortrait 2024-07-12 16:19:03 +09:00
docs 📄 [docs] Add comprehensive documentation in Japanese for LivePortrait 2024-07-12 16:17:03 +09:00
pretrained_weights feat: launch LivePortrait 2024-07-04 04:32:47 +08:00
src chore: format code 2024-07-12 14:34:28 +08:00
.gitignore feat: add several features (#89) 2024-07-10 23:39:49 +08:00
app.py feat: add several features (#89) 2024-07-10 23:39:49 +08:00
docker-compose.yml 🚀 [feat] Add Docker Compose setup for LivePortrait 2024-07-12 16:16:02 +09:00
inference.py chore: format code 2024-07-12 14:34:28 +08:00
LICENSE Initial commit 2024-07-04 00:15:03 +08:00
readme.md 🌐 [docs] Update main README with Docker Compose instructions and Japanese doc link 2024-07-12 16:17:17 +09:00
requirements.txt feat: add several features (#89) 2024-07-10 23:39:49 +08:00
speed.py feat: add several features (#89) 2024-07-10 23:39:49 +08:00

LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control

Pengfei Wan 1 Di Zhang 1
1 Kuaishou Technology 2 University of Science and Technology of China 3 Fudan University


JA doc

showcase
🔥 For more results, visit our homepage 🔥

🔥 Updates

  • 2024/07/10: 💪 We support audio and video concatenating, driving video auto-cropping, and template making to protect privacy. More to see here.
  • 2024/07/09: 🤗 We released the HuggingFace Space, thanks to the HF team and Gradio!
  • 2024/07/04: 😊 We released the initial version of the inference code and models. Continuous updates, stay tuned!
  • 2024/07/04: 🔥 We released the homepage and technical report on arXiv.

Introduction

This repo, named LivePortrait, contains the official PyTorch implementation of our paper LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control. We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.

🔥 Getting Started

1. Clone the code and prepare the environment

git clone https://github.com/KwaiVGI/LivePortrait
cd LivePortrait

# create env using conda
conda create -n LivePortrait python==3.9.18
conda activate LivePortrait
# install dependencies with pip
pip install -r requirements.txt

Note: make sure your system has FFmpeg installed!

2. Download pretrained weights

The easiest way to download the pretrained weights is from HuggingFace:

# you may need to run `git lfs install` first
git clone https://huggingface.co/KwaiVGI/liveportrait pretrained_weights

Alternatively, you can download all pretrained weights from Google Drive or Baidu Yun. Unzip and place them in ./pretrained_weights.

Ensuring the directory structure is as follows, or contains:

pretrained_weights
├── insightface
│   └── models
│       └── buffalo_l
│           ├── 2d106det.onnx
│           └── det_10g.onnx
└── liveportrait
    ├── base_models
    │   ├── appearance_feature_extractor.pth
    │   ├── motion_extractor.pth
    │   ├── spade_generator.pth
    │   └── warping_module.pth
    ├── landmark.onnx
    └── retargeting_models
        └── stitching_retargeting_module.pth

3. Inference 🚀

Fast hands-on

python inference.py

If the script runs successfully, you will get an output mp4 file named animations/s6--d0_concat.mp4. This file includes the following results: driving video, input image, and generated result.

image

Or, you can change the input by specifying the -s and -d arguments:

python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4

# disable pasting back to run faster
python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback

# more options to see
python inference.py -h

Driving video auto-cropping

📕 To use your own driving video, we recommend:

  • Crop it to a 1:1 aspect ratio (e.g., 512x512 or 256x256 pixels), or enable auto-cropping by --flag_crop_driving_video.
  • Focus on the head area, similar to the example videos.
  • Minimize shoulder movement.
  • Make sure the first frame of driving video is a frontal face with neutral expression.

Below is a auto-cropping case by --flag_crop_driving_video:

python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d13.mp4 --flag_crop_driving_video

If you find the results of auto-cropping is not well, you can modify the --scale_crop_video, --vy_ratio_crop_video options to adjust the scale and offset, or do it manually.

Motion template making

You can also use the auto-generated motion template files ending with .pkl to speed up inference, and protect privacy, such as:

python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d5.pkl

Discover more interesting results on our Homepage 😊

4. Gradio interface 🤗

We also provide a Gradio interface for a better experience, just run by:

python app.py

You can specify the --server_port, --share, --server_name arguments to satisfy your needs!

Or, try it out effortlessly on HuggingFace 🤗

5. Inference speed evaluation 🚀🚀🚀

We have also provided a script to evaluate the inference speed of each module:

python speed.py

Below are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with torch.compile:

Model Parameters(M) Model Size(MB) Inference(ms)
Appearance Feature Extractor 0.84 3.3 0.82
Motion Extractor 28.12 108 0.84
Spade Generator 55.37 212 7.59
Warping Module 45.53 174 5.21
Stitching and Retargeting Modules 0.23 2.3 0.31

Note: The values for the Stitching and Retargeting Modules represent the combined parameter counts and total inference time of three sequential MLP networks.

Community Resources 🤗

Discover the invaluable resources contributed by our community to enhance your LivePortrait experience:

And many more amazing contributions from our community!

Docker Compose Setup

You can also easily run the LivePortrait application using Docker Compose. The repository already includes a docker-compose.yml file. Follow these steps:

  1. Ensure you are in the root directory of the project.

  2. Start the application using Docker Compose:

docker-compose up

This will start the LivePortrait application inside a Docker container, accessible on port 8890.

Note

This configuration is set up to use an NVIDIA GPU. If you don't have a GPU available, you may need to adjust the deploy section in the docker-compose.yml file accordingly.

Once the application is running, you can access the Gradio interface by opening http://localhost:8890 in your web browser.

Acknowledgements

We would like to thank the contributors of FOMM, Open Facevid2vid, SPADE, InsightFace repositories, for their open research and contributions.

Citation 💖

If you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:

@article{guo2024liveportrait,
  title   = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},
  author  = {Guo, Jianzhu and Zhang, Dingyun and Liu, Xiaoqiang and Zhong, Zhizhou and Zhang, Yuan and Wan, Pengfei and Zhang, Di},
  journal = {arXiv preprint arXiv:2407.03168},
  year    = {2024}
}