Upload 2 files

README.md

@@ -1,3 +1,229 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+base_model:
+  - THUDM/CogVideoX-5b
+language:
+  - en
+tags:
+  - video-generation
+  - paddlemix
+---
+
+Englishh | [简体中文](README_zh.md)
+# VCtrl
+<p style="text-align: center;">
+  <p align="center"> 
+  <a href="https://huggingface.co/PaddleMIX">🤗 Huggingface Space</a> |
+  <a href="https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl">🌐 Github </a> | 
+  <a href="">📜 arxiv </a> |
+  <a href="https://pp-vctrl.github.io/">📷 Project </a> 
+</p>
+
+## Model Introduction
+**VCtrl** is a versatile video generation control model that introduces an auxiliary conditional encoder to flexibly connect with various control modules while avoiding large-scale retraining of the original generator. The model efficiently transmits control signals through sparse residual connections and standardizes diverse control inputs into a unified representation via a consistent encoding process. Task-specific masks are further incorporated to enhance adaptability. Thanks to this unified and flexible design, VCtrl can be widely applied in ​**character animation**, ​**scene transition**, ​**video editing**, and other video generation scenarios. The table below provides detailed information about the video generation models we offer:
+
+<table  style="border-collapse: collapse; width: 100%;">
+  <tr>
+    <th style="text-align: center;">Model Name</th>
+    <th style="text-align: center;">VCtrl-Canny</th>
+    <th style="text-align: center;">VCtrl-Mask</th>
+    <th style="text-align: center;">VCtrl-Pose</th>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Video Resolution</td>
+    <td colspan="1" style="text-align: center;">720 * 480</td>
+    <td colspan="1" style="text-align: center;"> 720 * 480 </td>
+    <td colspan="1 style="text-align: center;"> 720 * 480 & 480 * 720 </td>
+    </tr>
+  <tr>
+    <td style="text-align: center;">Inference Precision</td>
+    <td colspan="3" style="text-align: center;"><b>FP16(Recommended)</b></td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Single GPU VRAM Usage</td>
+    <td colspan="3"  style="text-align: center;"><b>V100: 32GB minimum*</b></td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Inference Speed<br>(Step = 25, FP16)</td>
+    <td colspan="3" style="text-align: center;">Single A100: ~300s(49 frames)<br>Single V100: ~400s(49 frames)</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Prompt Language</td>
+    <td colspan="5" style="text-align: center;">English*</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Prompt Length Limit</td>
+    <td colspan="3" style="text-align: center;">224 Tokens</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Video Length</td>
+    <td colspan="3" style="text-align: center;">T2V model supports only 49 frames, I2V model can extend to any frame count</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">Frame Rate</td>
+    <td colspan="3" style="text-align: center;">30 FPS </td>
+  </tr>
+</table>
+
+## Quick Start 🤗
+
+This model is now supported for deployment using the ppdiffusers library from paddlemix. Follow the steps below to get started.
+
+**We recommend visiting our [github](https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl) for a better experience.**
+
+1. Install dependencies
+
+```shell
+# Clone the PaddleMIX repository
+git clone https://github.com/PaddlePaddle/PaddleMIX.git
+# Install paddlemix
+cd PaddleMIX
+pip install -e .
+# Install ppdiffusers
+pip install -e ppdiffusers
+# Install paddlenlp
+pip install paddlenlp==v3.0.0-beta2
+# Navigate to the vctrl directory
+cd ppdiffusers/examples/ppvctrl
+# Install other required dependencies
+pip install -r requirements.txt
+# Install paddlex
+pip install paddlex==3.0.0b2
+```
+
+2. Run the code
+
+```python
+import os
+import paddle
+import numpy as np
+from decord import VideoReader
+from moviepy.editor import ImageSequenceClip
+from PIL import Image
+from ppdiffusers import (
+    CogVideoXDDIMScheduler,
+    CogVideoXTransformer3DVCtrlModel,
+    CogVideoXVCtrlPipeline,
+    VCtrlModel,
+)
+def write_mp4(video_path, samples, fps=8):
+    clip = ImageSequenceClip(samples, fps=fps)
+    clip.write_videofile(video_path, audio_codec="aac")
+
+
+def save_vid_side_by_side(batch_output, validation_control_images, output_folder, fps):
+    flattened_batch_output = [img for sublist in batch_output for img in sublist]
+    ori_video_path = output_folder + "/origin_predict.mp4"
+    video_path = output_folder + "/test_1.mp4"
+    ori_final_images = []
+    final_images = []
+    outputs = []
+
+    def get_concat_h(im1, im2):
+        dst = Image.new("RGB", (im1.width + im2.width, max(im1.height, im2.height)))
+        dst.paste(im1, (0, 0))
+        dst.paste(im2, (im1.width, 0))
+        return dst
+
+    for image_list in zip(validation_control_images, flattened_batch_output):
+        predict_img = image_list[1].resize(image_list[0].size)
+        result = get_concat_h(image_list[0], predict_img)
+        ori_final_images.append(np.array(image_list[1]))
+        final_images.append(np.array(result))
+        outputs.append(np.array(predict_img))
+    write_mp4(ori_video_path, ori_final_images, fps=fps)
+    write_mp4(video_path, final_images, fps=fps)
+    output_path = output_folder + "/output.mp4"
+    write_mp4(output_path, outputs, fps=fps)
+
+
+def load_images_from_folder_to_pil(folder):
+    images = []
+    valid_extensions = {".jpg", ".jpeg", ".png", ".bmp", ".gif", ".tiff"}
+
+    def frame_number(filename):
+        new_pattern_match = re.search("frame_(\\d+)_7fps", filename)
+        if new_pattern_match:
+            return int(new_pattern_match.group(1))
+        matches = re.findall("\\d+", filename)
+        if matches:
+            if matches[-1] == "0000" and len(matches) > 1:
+                return int(matches[-2])
+            return int(matches[-1])
+        return float("inf")
+
+    sorted_files = sorted(os.listdir(folder), key=frame_number)
+    for filename in sorted_files:
+        ext = os.path.splitext(filename)[1].lower()
+        if ext in valid_extensions:
+            img = Image.open(os.path.join(folder, filename)).convert("RGB")
+            images.append(img)
+    return images
+
+
+def load_images_from_video_to_pil(video_path):
+    images = []
+    vr = VideoReader(video_path)
+    length = len(vr)
+    for idx in range(length):
+        frame = vr[idx].asnumpy()
+        images.append(Image.fromarray(frame))
+    return images
+
+
+validation_control_images = load_images_from_video_to_pil('your_path')
+prompt = 'Group of fishes swimming in aquarium.'
+vctrl = VCtrlModel.from_pretrained(
+            paddlemix/vctrl-5b-t2v-canny,
+            low_cpu_mem_usage=True,
+            paddle_dtype=paddle.float16
+        )
+pipeline = CogVideoXVCtrlPipeline.from_pretrained(
+            paddlemix/cogvideox-5b-vctrl, 
+            vctrl=vctrl, 
+            paddle_dtype=paddle.float16, 
+            low_cpu_mem_usage=True,
+            map_location="cpu",
+        )
+pipeline.scheduler = CogVideoXDDIMScheduler.from_config(pipeline.scheduler.config, timestep_spacing="trailing")
+pipeline.vae.enable_tiling()
+pipeline.vae.enable_slicing()
+task='canny'
+final_result=[]
+video = pipeline(
+        prompt=prompt,
+        num_inference_steps=25,
+        num_frames=49,
+        guidance_scale=35,
+        generator=paddle.Generator().manual_seed(42),
+        conditioning_frames=validation_control_images[:num_frames],
+        conditioning_frame_indices=list(range(num_frames)),
+        conditioning_scale=1.0,
+        width=720,
+        height=480,
+        task='canny',
+        conditioning_masks=validation_mask_images[:num_frames] if task == "mask" else None,
+        vctrl_layout_type='spacing',
+    ).frames[0]
+    final_result.append(video)
+    save_vid_side_by_side(final_result, validation_control_images[:num_frames], 'save.mp4', fps=30)
+```
+
+## In-Depth Exploration
+
+Welcome to our[github]("https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl")， where you will find：
+
+1. More detailed technical explanations and code walkthroughs.
+2. Algorithm details for extracting control conditions.
+3. Detailed code for model inference.
+4. Project update logs and more interactive opportunities.
+5. PaddleMix toolchain to help you better utilize the model.
+
+## Citation
+
+```
+@article{yang2024cogvideox,
+  title={VCtrl: Enabling Versatile Controls for Video Diffusion Models},
+  year={2025}
+}
+```

README_zh.md

@@ -0,0 +1,230 @@
+---
+license: apache-2.0
+base_model:
+  - THUDM/CogVideoX-5b
+language:
+  - en
+tags:
+  - video-generation
+  - paddlemix
+---
+
+简体中文 | [English](README.md)
+# VCtrl
+<p style="text-align: center;">
+  <p align="center"> 
+  <a href="https://huggingface.co/PaddleMIX">🤗 Huggingface Space</a> |
+  <a href="https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl">🌐 Github </a> | 
+  <a href="">📜 arxiv </a> |
+  <a href="https://pp-vctrl.github.io/">📷 Project </a> 
+</p>
+
+## 模型介绍
+**VCtrl** 是一个通用的视频生成控制模型，通过引入辅助条件编码器，能够灵活对接各类控制模块，并且在不改变原始生成器的前提下避免了大规模重训练。该模型利用稀疏残差连接实现对控制信号的高效传递，同时通过统一的条件编码流程，将多种控制输入转换为标准化表示，再结合任务特定掩码以提升适应性。得益于这种统一而灵活的设计，VCtrl 可广泛应用于**人物动画**、**场景转换**、**视频编辑**等视频生成场景。下表展示我们在本代提供的视频生成模型列表相关信息:
+
+<table  style="border-collapse: collapse; width: 100%;">
+  <tr>
+    <th style="text-align: center;">模型名</th>
+    <th style="text-align: center;">VCtrl-Canny</th>
+    <th style="text-align: center;">VCtrl-Mask</th>
+    <th style="text-align: center;">VCtrl-Pose</th>
+  </tr>
+  <tr>
+    <td style="text-align: center;">视频分辨率</td>
+    <td colspan="1" style="text-align: center;">720 * 480</td>
+    <td colspan="1" style="text-align: center;"> 720*480 </td>
+    <td colspan="1" style="text-align: center;"> 720*480 & 480*720 </td>
+    </tr>
+  <tr>
+    <td style="text-align: center;">推理精度</td>
+    <td colspan="3" style="text-align: center;"><b>FP16(推荐)</b></td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">单GPU显存消耗</td>
+    <td colspan="3"  style="text-align: center;"><b>V100: 32GB minimum*</b></td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">推理速度<br>(Step = 25, FP16)</td>
+    <td colspan="3" style="text-align: center;">单卡A100: ~300秒(49帧)<br>单卡V100: ~400秒(49帧)</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">提示词语言</td>
+    <td colspan="5" style="text-align: center;">English*</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">提示词长度上限</td>
+    <td colspan="3" style="text-align: center;">224 Tokens</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">视频长度</td>
+    <td colspan="3" style="text-align: center;">T2V模型只支持49帧，I2V模型可以扩展为任意帧</td>
+  </tr>
+  <tr>
+    <td style="text-align: center;">帧率</td>
+    <td colspan="3" style="text-align: center;">30 帧 / 秒 </td>
+  </tr>
+</table>
+
+##  快速开始 🤗
+
+本模型已经支持使用 paddlemix 的 ppdiffusers 库进行部署，你可以按照以下步骤进行部署。
+
+**我们推荐您进入我们的 [github](https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl)以获得更好的体验。**
+
+1. 安装对应的依赖
+
+```shell
+# 克隆 PaddleMIX 仓库
+git clone https://github.com/PaddlePaddle/PaddleMIX.git
+#安装paddlemix
+cd PaddleMIX
+pip install -e .
+# 安装ppdiffusers
+pip install -e ppdiffusers
+# 安装paddlenlp
+pip install paddlenlp==v3.0.0-beta2
+# 进入 vctrl目录
+cd ppdiffusers/examples/ppvctrl
+# 安装其他所需的依赖
+pip install -r requirements.txt
+#安装paddlex
+pip install paddlex==3.0.0b2
+
+```
+
+2. 运行代码
+
+```python
+import os
+import paddle
+import numpy as np
+from decord import VideoReader
+from moviepy.editor import ImageSequenceClip
+from PIL import Image
+from ppdiffusers import (
+    CogVideoXDDIMScheduler,
+    CogVideoXTransformer3DVCtrlModel,
+    CogVideoXVCtrlPipeline,
+    VCtrlModel,
+)
+def write_mp4(video_path, samples, fps=8):
+    clip = ImageSequenceClip(samples, fps=fps)
+    clip.write_videofile(video_path, audio_codec="aac")
+
+
+def save_vid_side_by_side(batch_output, validation_control_images, output_folder, fps):
+    flattened_batch_output = [img for sublist in batch_output for img in sublist]
+    ori_video_path = output_folder + "/origin_predict.mp4"
+    video_path = output_folder + "/test_1.mp4"
+    ori_final_images = []
+    final_images = []
+    outputs = []
+
+    def get_concat_h(im1, im2):
+        dst = Image.new("RGB", (im1.width + im2.width, max(im1.height, im2.height)))
+        dst.paste(im1, (0, 0))
+        dst.paste(im2, (im1.width, 0))
+        return dst
+
+    for image_list in zip(validation_control_images, flattened_batch_output):
+        predict_img = image_list[1].resize(image_list[0].size)
+        result = get_concat_h(image_list[0], predict_img)
+        ori_final_images.append(np.array(image_list[1]))
+        final_images.append(np.array(result))
+        outputs.append(np.array(predict_img))
+    write_mp4(ori_video_path, ori_final_images, fps=fps)
+    write_mp4(video_path, final_images, fps=fps)
+    output_path = output_folder + "/output.mp4"
+    write_mp4(output_path, outputs, fps=fps)
+
+
+def load_images_from_folder_to_pil(folder):
+    images = []
+    valid_extensions = {".jpg", ".jpeg", ".png", ".bmp", ".gif", ".tiff"}
+
+    def frame_number(filename):
+        new_pattern_match = re.search("frame_(\\d+)_7fps", filename)
+        if new_pattern_match:
+            return int(new_pattern_match.group(1))
+        matches = re.findall("\\d+", filename)
+        if matches:
+            if matches[-1] == "0000" and len(matches) > 1:
+                return int(matches[-2])
+            return int(matches[-1])
+        return float("inf")
+
+    sorted_files = sorted(os.listdir(folder), key=frame_number)
+    for filename in sorted_files:
+        ext = os.path.splitext(filename)[1].lower()
+        if ext in valid_extensions:
+            img = Image.open(os.path.join(folder, filename)).convert("RGB")
+            images.append(img)
+    return images
+
+
+def load_images_from_video_to_pil(video_path):
+    images = []
+    vr = VideoReader(video_path)
+    length = len(vr)
+    for idx in range(length):
+        frame = vr[idx].asnumpy()
+        images.append(Image.fromarray(frame))
+    return images
+
+
+validation_control_images = load_images_from_video_to_pil('your_path')
+prompt = 'Group of fishes swimming in aquarium.'
+vctrl = VCtrlModel.from_pretrained(
+            paddlemix/vctrl-5b-t2v-canny,
+            low_cpu_mem_usage=True,
+            paddle_dtype=paddle.float16
+        )
+pipeline = CogVideoXVCtrlPipeline.from_pretrained(
+            paddlemix/cogvideox-5b-vctrl, 
+            vctrl=vctrl, 
+            paddle_dtype=paddle.float16, 
+            low_cpu_mem_usage=True,
+            map_location="cpu",
+        )
+pipeline.scheduler = CogVideoXDDIMScheduler.from_config(pipeline.scheduler.config, timestep_spacing="trailing")
+pipeline.vae.enable_tiling()
+pipeline.vae.enable_slicing()
+task='canny'
+final_result=[]
+video = pipeline(
+        prompt=prompt,
+        num_inference_steps=25,
+        num_frames=49,
+        guidance_scale=35,
+        generator=paddle.Generator().manual_seed(42),
+        conditioning_frames=validation_control_images[:num_frames],
+        conditioning_frame_indices=list(range(num_frames)),
+        conditioning_scale=1.0,
+        width=720,
+        height=480,
+        task='canny',
+        conditioning_masks=validation_mask_images[:num_frames] if task == "mask" else None,
+        vctrl_layout_type='spacing',
+    ).frames[0]
+    final_result.append(video)
+    save_vid_side_by_side(final_result, validation_control_images[:num_frames], 'save.mp4', fps=30)
+```
+
+## 深入研究
+
+欢迎进入我们的 [github]("https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl")，你将获得：
+
+1. 更加详细的技术细节介绍和代码解释。
+2. 控制条件的提取算法细节。
+3. 模型推理的详细代码。
+4. 项目更新日志动态，更多互动机会。
+5. PaddleMix工具链，帮助您更好的使用模型。
+
+## 引用
+
+```
+@article{yang2024cogvideox,
+  title={VCtrl: Enabling Versatile Controls for Video Diffusion Models},
+  year={2025}
+}
+```