💻 How to Inference & Test Metrics (FID, CLIP Score, GenEval, DPG-Bench, etc...)

This ToolKit will automatically inference your model and log the metrics results onto wandb as chart for better illustration. We curerntly support:

0. Install corresponding env for GenEval and DPG-Bench

Make sure you can activate the following envs:

conda activate geneval(GenEval)
conda activate dpg(DGB-Bench)

0.1 Prepare data.

Metirc FID & CLIP-Score on MJHQ-30K

from huggingface_hub import hf_hub_download

hf_hub_download(
  repo_id="playgroundai/MJHQ-30K",
  filename="mjhq30k_imgs.zip",
  local_dir="data/test/PG-eval-data/MJHQ-30K/",
  repo_type="dataset"
)

Unzip mjhq30k_imgs.zip into its per-category folder structure.

data/test/PG-eval-data/MJHQ-30K/imgs/
├── animals
├── art
├── fashion
├── food
├── indoor
├── landscape
├── logo
├── people
├── plants
└── vehicles

0.2 Prepare checkpoints

huggingface-cli download  Efficient-Large-Model/Sana_1600M_1024px --repo-type model --local-dir ./output/Sana_1600M_1024px --local-dir-use-symlinks False

1. directly [Inference and Metric] a .pth file

# We provide four scripts for evaluating metrics:
fid_clipscore_launch=scripts/bash_run_inference_metric.sh
geneval_launch=scripts/bash_run_inference_metric_geneval.sh
dpg_launch=scripts/bash_run_inference_metric_dpg.sh
image_reward_launch=scripts/bash_run_inference_metric_imagereward.sh

# Use following format to metric your models:
# bash $correspoinding_metric_launch $your_config_file_path $your_relative_pth_file_path

# example
bash $geneval_launch \
    configs/sana_config/1024ms/Sana_1600M_img1024.yaml \
    output/Sana_1600M_1024px/checkpoints/Sana_1600M_1024px.pth

2. [Inference and Metric] a list of .pth files using a txt file

You can also write all your pth files of a job in one txt file, eg. model_paths.txt

# Use following format to metric your models, gathering in a txt file:
# bash $correspoinding_metric_launch $your_config_file_path $your_txt_file_path_containing_pth_path

# We suggest follow the file tree structure in our project for robust experiment
# example
bash scripts/bash_run_inference_metric.sh \
    configs/sana_config/1024ms/Sana_1600M_img1024.yaml \
    asset/model_paths.txt

3. You will get the following data tree.

output
├──your_job_name/  (everything will be saved here)
│  ├──config.yaml
│  ├──train_log.log

│  ├──checkpoints    (all checkpoints)
│  │  ├──epoch_1_step_6666.pth
│  │  ├──epoch_1_step_8888.pth
│  │  ├──......

│  ├──vis    (all visualization result dirs)
│  │  ├──visualization_file_name
│  │  │  ├──xxxxxxx.jpg
│  │  │  ├──......
│  │  ├──visualization_file_name2
│  │  │  ├──xxxxxxx.jpg
│  │  │  ├──......
│  ├──......

│  ├──metrics    (all metrics testing related files)
│  │  ├──model_paths.txt  Optional(👈)(relative path of testing ckpts)
│  │  │  ├──output/your_job_name/checkpoings/epoch_1_step_6666.pth
│  │  │  ├──output/your_job_name/checkpoings/epoch_1_step_8888.pth
│  │  ├──fid_img_paths.txt  Optional(👈)(name of testing img_dir in vis)
│  │  │  ├──visualization_file_name
│  │  │  ├──visualization_file_name2
│  │  ├──cached_img_paths.txt  Optional(👈)
│  │  ├──......