adding ContraCLIP folder

ContraCLIP/.gitignore

@@ -0,0 +1,21 @@
+.directory
+*/.directory
+*~
+.idea/
+contra-clip-venv/
+*.pyc
+__pycache__/
+*/__pycache__/
+dev/
+notebooks/
+figs/inkscape/
+models/pretrained/
+
+scripts/train/BACKUP/
+scripts/eval/BACKUP/
+scripts/compare/BACKUP/
+
+!experiments/
+experiments/*
+experiments/latent_codes/TMP/
+!experiments/latent_codes/

ContraCLIP/README.md

@@ -0,0 +1,178 @@
+# ContraCLIP: Interpretable GAN generation driven by pairs of contrasting sentences
+
+Authors official PyTorch implementation of the **[ContraCLIP: Interpretable GAN generation driven by pairs of contrasting sentences](https://arxiv.org/pdf/2206.02104.pdf)**. If you use this code for your research, please [**cite**](#citation) our paper.
+
+> **ContraCLIP: Interpretable GAN generation driven by pairs of contrasting sentences**<br>
+> Christos Tzelepis, James Oldfield, Georgios Tzimiropoulos, and Ioannis Patras<br>
+> https://arxiv.org/abs/2206.02104 <br>
+> ![ContraCLIP Summary](figs/summary.png)
+>
+> **Abstract**: This work addresses the problem of discovering non-linear interpretable paths in the latent space of pre-trained GANs in a model-agnostic manner. In the proposed method, the discovery is driven by a set of pairs of natural language sentences with contrasting semantics, named semantic dipoles, that serve as the limits of the interpretation that we require by the trainable latent paths to encode. By using the pre-trained CLIP encoder, the sentences are projected into the vision-language space, where they serve as dipoles, and where RBF-based warping functions define a set of non-linear directional paths, one for each semantic dipole, allowing in this way traversals from one semantic pole to the other. By defining an objective that discovers paths in the latent space of GANs that generate changes along the desired paths in the vision-language embedding space, we provide an intuitive way of controlling the underlying generating factors and address some of the limitations of the state-of-the-art works, namely, that a) they are typically tailored to specific GAN architectures (i.e., StyleGAN), b) they disregard the relative position of the manipulated and the original image in the image embedding and the relative position of the image and the text embeddings, and c) they lead to abrupt image manipulations and quickly arrive at regions of low density and, thus, low image quality, providing limited control of the generative factors.
+
+
+| Semantic Dipole (i.e., contrasting sentences given in natural language) |                           Example                            |
+| ------------------------------------------------------------ | :----------------------------------------------------------: |
+| *"a picture of an **angry shaved man**." &#8594; "a picture of a **man** with a **beard crying**."* <br>[StyleGAN2@FFHQ] | <img src="figs/examples/stylegan2ffhq_angryshaved2beardcrying.gif" width="500"/> |
+| *"a picture of a person with **open eyes**." &#8594; "a picture of a person with **closed eyes**."* <br>[StyleGAN2@FFHQ] | <img src="figs/examples/stylegan2ffhq_eyes.gif" width="500"/> |
+| *"a picture of a **young person**." &#8594; "a picture of an **old person**."* <br>[StyleGAN2@FFHQ] | <img src="figs/examples/stylegan2ffhq_young2old.gif" width="500"/> |
+| *"a picture of a **man** with **hair**." &#8594; "a picture of a **bald man**."* <br>[ProgGAN@CelebA-HQ] | <img src="figs/examples/pggancelebahq_hair2bald.gif" width="500"/> |
+| *"a picture of a person with **happy** face." &#8594; "a picture of a person with **surprised** face."* <br>[ProgGAN@CelebA-HQ] | <img src="figs/examples/pggancelebahq_happy2surprised.gif" width="500"/> |
+| *"a picture of a **face without makeup**." &#8594; "a picture of a **face with makeup**."* <br>[ProgGAN@CelebA-HQ] | <img src="figs/examples/pggancelebahq_makeup.gif" width="500"/> |
+| *"a picture of an **ugly cat**." &#8594; "a picture of a **cute cat**."* <br>[StyleGAN2@AFHQ-Cats] | <img src="figs/examples/stylegan2afhqcats_ugly2cute.gif" width="500"/> |
+| *"a picture of a **dog** with **small eyes**." &#8594; "a picture of a **dog** with **big eyes**."* <br>[StyleGAN2@AFHQ-Dogs] | <img src="figs/examples/stylegan2afhqdogs_smalleyes2bigeyes.gif" width="500"/> |
+
+
+
+## Overview
+
+![ContraCLIP Overview](./figs/overview.svg)
+<p alighn="center">
+The CLIP text space, warped due to semantic dipoles of contrasting pairs of sentences in natural language, provides supervision to the optimisation of non-linear interpretable paths in the latent space of a pre-trained GAN.
+</p>
+
+
+## Installation
+
+We recommend installing the required packages using python's native virtual environment as follows:
+
+```bash
+$ python -m venv contra-clip-venv
+$ source contra-clip-venv/bin/activate
+(contra-clip-venv) $ pip install --upgrade pip
+(contra-clip-venv) $ pip install -r requirements.txt
+(contra-clip-venv) $ pip install git+https://github.com/openai/CLIP.git
+(contra-clip-venv) $ pip install --pre torch torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cu113
+```
+
+For using the aforementioned virtual environment in a Jupyter Notebook, you need to manually add the kernel as follows:
+
+```bash
+(contra-clip-venv) $ python -m ipykernel install --user --name=contra-clip-venv
+```
+
+
+
+## Prerequisite pre-trained models and pre-trained ContraCLIP models
+
+Download the prerequisite pre-trained models (GAN generators and various pre-trained detectors, such as ArcFace, FairFace, etc), as well as (optionally) pre-trained ContraCLIP models (by passing `-m` or `----contraclip-models`) as follows:
+
+```bash
+(contra-clip-venv) $ python download.py -m
+```
+
+This will create a directory `models/pretrained` with the following sub-directories (~3.3 GiB):
+```
+./models/pretrained/
+├── genforce
+│   ├── pggan_car256.pth
+│   ├── pggan_celebahq1024.pth
+│   ├── pggan_church256.pth
+│   ├── stylegan2_afhqcat512.pth
+│   ├── stylegan2_afhqdog512.pth
+│   ├── stylegan2_car512.pth
+│   ├── stylegan2_church256.pth
+│   └── stylegan2_ffhq1024.pth
+├── arcface
+│   └── model_ir_se50.pth
+├── au_detector
+│   └── disfa_adaptation_f0.pth
+├── celeba_attributes
+│   └── eval_predictor.pth.tar
+├── fairface
+│   ├── fairface_alldata_4race_20191111.pt
+│   └── res34_fair_align_multi_7_20190809.pt
+├── hopenet
+│   ├── hopenet_alpha1.pkl
+│   ├── hopenet_alpha2.pkl
+│   └── hopenet_robust_alpha1.pkl
+└── sfd
+    └── s3fd-619a316812.pth
+```
+
+as well as, a directory `experiments/complete/` (if not already created by the user upon an experiment's completion) for downloading the ContraCLIP pre-trained models with the following sub-directories (~160 MiB):
+
+```
+.experiments/complete/
+├── ContraCLIP_pggan_celebahq1024-Z-K9-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-attributes
+├── ContraCLIP_pggan_celebahq1024-Z-K9-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-cossim-20000-attributes
+├── ContraCLIP_stylegan2_afhqcat512-W+-K3-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-cats
+├── ContraCLIP_stylegan2_afhqdog512-W+-K4-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-dogs
+├── ContraCLIP_stylegan2_car512-W+-K3-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-cars
+├── ContraCLIP_stylegan2_ffhq1024-W+-K21-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-expressions
+├── ContraCLIP_stylegan2_ffhq1024-W+-K21-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-cossim-20000-expressions
+├── ContraCLIP_stylegan2_ffhq1024-W+-K3-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-complex
+├── ContraCLIP_stylegan2_ffhq1024-W+-K3-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-expressions3
+├── ContraCLIP_stylegan2_ffhq1024-W+-K3-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-cossim-20000-complex
+├── ContraCLIP_stylegan2_ffhq1024-W+-K3-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-cossim-20000-expressions3
+├── ContraCLIP_stylegan2_ffhq1024-W+-K9-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-contrastive_0.07-20000-attributes
+└── ContraCLIP_stylegan2_ffhq1024-W+-K9-D64-lss_beta_0.5-eps0.1_0.2-nonlinear_css_beta_0.5-cossim-20000-attributes
+```
+
+We note that the pre-trained detectors (such as ArcFace) are used only during the evaluation stage (**no ID preserving loss is imposed during training**).
+
+
+
+## Training
+
+For training a ContraCLIP model you need to use `train.py` (check its basic usage by running  `python train.py -h`). For example, in order to train a ContraCLIP model for the corpus of contrasting sentences called "expressions3" (defined in `lib/config.py`)  on the StyleGAN2 pre-trained (on FFHQ) generator (in its `W` latent space with a truncation parameter equal to `0.7`), the following command:
+
+```bash
+(contra-clip-venv) $ python train.py --gan=stylegan2_ffhq1024 --truncation=0.7 --stylegan-space=W --corpus=expressions3 --num-latent-support-dipoles=128 --loss=contrastive --temperature=0.5 --beta=0.75 --min-shift-magnitude=0.1 --max-shift-magnitude=0.2 --batch-size=3 --max-iter=120000 --log-freq=10--ckp-freq=100
+```
+
+In the example above, the batch size is set to `3` and the training will be conducted for `120000` iterations. Minimum and maximum shift magnitudes are set to `0.1` and `0.2`, respectively, and the number of support dipoles for each latent path is set to `128` (please see the [WarpedGANSpace](https://github.com/chi0tzp/WarpedGANSpace) for more details). Moreover, `contrastive` loss is being used with a temperature parameter equal to `0.5`. The `beta` parameter  of the CLIP text space RBF dipoles is set to `0.75`. A set of auxiliary training scripts (for the results reported in the paper) can be found under `scripts/train/`.
+
+The training script will create a directory with the following name format:
+
+```
+ContraCLIP_<gan_type>-<latent_space>-K<num_of_paths>-D<num_latent_support_sets>-eps<min_shift_magnitude>_<max_shift_magnitude>-<linear|nonlinear>_beta-<beta>-contrastive_<temperature>-<corpus>
+```
+
+For instance, `ContraCLIP_stylegan2_ffhq1024-W-K3-D128-eps0.1_0.2-nonlinear_beta-0.75-contrastive_0.5-expressions3`, under `experiments/wip/` while training is in progress, which after training completion, will be copied under `experiments/complete/`. This directory has the following structure:
+
+```
+├── models/
+├── args.json
+├── stats.json
+└── command.sh
+```
+
+where `models/` contains the weights for the latent support sets (`latent_support_sets.pt`). While training is in progress (i.e., while this directory is found under `experiments/wip/`), the corresponding `models/` directory contains a checkpoint file (`checkpoint.pt`) containing the last iteration, and the weights for the latent support sets, so as to resume training. Re-run the same command, and if the last iteration is less than the given maximum number of iterations, training will resume from the last iteration. This directory will be referred to as `EXP_DIR` for the rest of this document. 
+
+
+
+## Evaluation
+
+As soon as a *ContraCLIP* model is trained, the corresponding experiment's directory (i.e., `EXP_DIR`) can be found under `experiments/complete/`. In order to evaluate the model, we can generate image sequences across the discovered latent paths (for the given pairs of contrasting sentences). For doing so, we need to create a pool of latent codes/images for the corresponding GAN type. This can be done using `sample_gan.py`. The pool of latent codes/images will be stored under `experiments/latent_codes/<gan_type>/`.  We will be referring to it as  `POOL` for the rest of this document. 
+
+For example, the following command will create a pool named `stylegan2_ffhq1024-4` under `experiments/latent_codes/stylegan2_ffhq1024/`:
+
+```bash
+(contra-clip-venv) $ python sample_gan.py -v --gan-type=stylegan2_ffhq1024 --stylegan-space=W --truncation=0.7 --num-samples=4
+```
+
+Latent space traversals can then be calculated using the script `traverse_latent_space.py` (please check its basic usage by running `traverse_latent_space.py -h`) for a given model and a given `POOL`. Upon completion, results (i.e., latent traversals) will be stored under the following directory:
+
+`experiments/complete/EXP_DIR/results/POOL/<2*shift_steps>_<eps>_<total_length>`,
+
+where `eps`,  `shift_steps`, and `total_length` denote respectively the shift magnitude (of a single step on the path), the number of such steps, and the total traversal length. A set of auxiliary evaluation scripts (for the results reported in the paper) can be found under `scripts/eval/`.
+
+
+
+## Citation
+
+```bibtex
+@misc{tzelepis2022contraclip,
+    author = {Tzelepis, Christos and James, Oldfield and Tzimiropoulos, Georgios and Patras, Ioannis},
+    title = {{ContraCLIP}: Interpretable {GAN} generation driven by pairs of contrasting sentences},
+    year={2022},
+    eprint={2206.02104},
+    archivePrefix={arXiv},
+    primaryClass={cs.CV}
+}
+```
+
+
+
+<!--Acknowledgement: This research was supported by the EU's Horizon 2020 programme H2020-951911 [AI4Media](https://www.ai4media.eu/) project.-->
+

ContraCLIP/calculate_jung_radii.py

@@ -0,0 +1,210 @@
+import argparse
+import numpy as np
+import os.path as osp
+import torch
+from lib import GENFORCE_MODELS
+from models.load_generator import load_generator
+from sklearn import linear_model
+from collections import defaultdict
+from tqdm import tqdm
+import json
+
+
+def make_dict():
+    return defaultdict(make_dict)
+
+
+def main():
+    """A script for calculating the radii of minimal enclosing balls for the latent space of a (i.e., in Z/W/W+ space),
+    given a truncation parameter. When applicable, a linear model is trained in order to predict the radii of the latent
+    codes, given a truncation parameter.
+
+    The parameters of the linear model (i.e., the weight w and the bias b) are stored for each GAN type and each latent
+    space in a json file (i.e., models/jung_radii.json) as a dictionary with the following format:
+        {
+            ...
+            <gan>:
+                {
+                    'Z': (<w>, <b>),
+                    'W':
+                        {
+                            ...
+                            <stylegan-layer>: (<w>, <b>),
+                            ...
+                        },
+                },
+            ...
+        }
+    so as, given a truncation parameter t, the radius is given as `w * t + b`.
+
+    Options:
+        -v, --verbose    : set verbose mode on
+        --num-samples    : set the number of latent codes to sample for generating images
+        --cuda           : use CUDA (default)
+        --no-cuda        : do not use CUDA
+    """
+    parser = argparse.ArgumentParser(description="Fit a linear model for the jung radius of GAN's latent code given "
+                                                 "a truncation parameter")
+    parser.add_argument('-v', '--verbose', action='store_true', help="verbose mode on")
+    parser.add_argument('--num-samples', type=int, default=1000, help="set number of latent codes to sample")
+    parser.add_argument('--cuda', dest='cuda', action='store_true', help="use CUDA during training")
+    parser.add_argument('--no-cuda', dest='cuda', action='store_false', help="do NOT use CUDA during training")
+    parser.set_defaults(cuda=True)
+    # ================================================================================================================ #
+
+    # Parse given arguments
+    args = parser.parse_args()
+
+    # CUDA
+    use_cuda = False
+    if torch.cuda.is_available():
+        if args.cuda:
+            use_cuda = True
+            torch.set_default_tensor_type('torch.cuda.FloatTensor')
+        else:
+            print("*** WARNING ***: It looks like you have a CUDA device, but aren't using CUDA.\n"
+                  "                 Run with --cuda for optimal training speed.")
+            torch.set_default_tensor_type('torch.FloatTensor')
+    else:
+        torch.set_default_tensor_type('torch.FloatTensor')
+
+    # Build jung radii dictionary and populate it
+    nested_dict = lambda: defaultdict(nested_dict)
+    jung_radii_dict = nested_dict()
+    for gan in GENFORCE_MODELS.keys():
+        ################################################################################################################
+        ##                                                                                                            ##
+        ##                                               [ StyleGANs ]                                                ##
+        ##                                                                                                            ##
+        ################################################################################################################
+        if 'stylegan' in gan:
+            ############################################################################################################
+            ##                                                                                                        ##
+            ##                                         [ StyleGAN / Z-space ]                                         ##
+            ##                                                                                                        ##
+            ############################################################################################################
+            # Build GAN generator model and load with pre-trained weights
+            if args.verbose:
+                print("  \\__Build GAN generator model G and load with pre-trained weights...")
+                print("      \\__GAN generator : {} (res: {})".format(gan, GENFORCE_MODELS[gan][1]))
+                print("      \\__Pre-trained weights: {}".format(GENFORCE_MODELS[gan][0]))
+
+            G = load_generator(model_name=gan, latent_is_w=False, verbose=args.verbose).eval()
+
+            # Upload GAN generator model to GPU
+            if use_cuda:
+                G = G.cuda()
+
+            # Latent codes sampling
+            if args.verbose:
+                print("  \\__Sample {} {}-dimensional latent codes...".format(args.num_samples, G.dim_z))
+            zs = torch.randn(args.num_samples, G.dim_z)
+
+            if use_cuda:
+                zs = zs.cuda()
+
+            # Calculate expected latent norm
+            if args.verbose:
+                print("  \\__Calculate Jung radius...")
+            jung_radius = torch.cdist(zs, zs).max() * np.sqrt(G.dim_z / (2 * (G.dim_z + 1)))
+            jung_radii_dict[gan]['Z'] = (0.0, jung_radius.cpu().detach().item())
+
+            ############################################################################################################
+            ##                                                                                                        ##
+            ##                                       [ StyleGAN / W/W+-space ]                                        ##
+            ##                                                                                                        ##
+            ############################################################################################################
+            # Build GAN generator model and load with pre-trained weights
+            if args.verbose:
+                print("  \\__Build GAN generator model G and load with pre-trained weights...")
+                print("      \\__GAN generator : {} (res: {})".format(gan, GENFORCE_MODELS[gan][1]))
+                print("      \\__Pre-trained weights: {}".format(GENFORCE_MODELS[gan][0]))
+
+            G = load_generator(model_name=gan, latent_is_w=True, verbose=args.verbose).eval()
+
+            # Upload GAN generator model to GPU
+            if use_cuda:
+                G = G.cuda()
+
+            # Latent codes sampling
+            if args.verbose:
+                print("  \\__Sample {} {}-dimensional latent codes...".format(args.num_samples, G.dim_z))
+            zs = torch.randn(args.num_samples, G.dim_z)
+
+            if use_cuda:
+                zs = zs.cuda()
+
+            # Get number of W layers for the given StyleGAN
+            stylegan_num_layers = G.get_w(zs, truncation=1.0).shape[1]
+
+            # Calculate expected latent norm and fit a linear model for each version of the W+ space
+            if args.verbose:
+                print("  \\__Calculate Jung radii and fit linear models...")
+            data_per_layer = dict()
+            tmp = []
+            for truncation in tqdm(np.linspace(0.1, 1.0, 100), desc="  \\__Calculate radii (W space): "):
+                ws = G.get_w(zs, truncation=truncation)[:, 0, :]
+                jung_radius = torch.cdist(ws, ws).max() * np.sqrt(ws.shape[1] / (2 * (ws.shape[1] + 1)))
+                tmp.append([truncation, jung_radius.cpu().detach().item()])
+            data_per_layer.update({0: tmp})
+
+            for ll in tqdm(range(1, stylegan_num_layers), desc="  \\__Calculate radii (W+ space): "):
+                tmp = []
+                for truncation in np.linspace(0.1, 1.0, 100):
+                    ws_plus = G.get_w(zs, truncation=truncation)[:, :ll + 1, :]
+                    ws_plus = ws_plus.reshape(ws_plus.shape[0], -1)
+                    jung_radius = torch.cdist(ws_plus, ws_plus).max() * \
+                        np.sqrt(ws_plus.shape[1] / (2 * (ws_plus.shape[1] + 1)))
+                    tmp.append([truncation, jung_radius.cpu().detach().item()])
+                data_per_layer.update({ll: tmp})
+
+            for ll, v in tqdm(data_per_layer.items(), desc="  \\__Fit linear models"):
+                v = np.array(v)
+                lm = linear_model.LinearRegression()
+                lm.fit(v[:, 0].reshape(-1, 1), v[:, 1].reshape(-1, 1))
+                jung_radii_dict[gan]['W'][ll] = (float(lm.coef_[0, 0]), float(lm.intercept_[0]))
+
+        ################################################################################################################
+        ##                                                                                                            ##
+        ##                                                [ ProgGAN ]                                                 ##
+        ##                                                                                                            ##
+        ################################################################################################################
+        else:
+            # Build GAN generator model and load with pre-trained weights
+            if args.verbose:
+                print("  \\__Build GAN generator model G and load with pre-trained weights...")
+                print("      \\__GAN generator : {} (res: {})".format(gan, GENFORCE_MODELS[gan][1]))
+                print("      \\__Pre-trained weights: {}".format(GENFORCE_MODELS[gan][0]))
+
+            G = load_generator(model_name=gan, latent_is_w=False, verbose=args.verbose).eval()
+
+            # Upload GAN generator model to GPU
+            if use_cuda:
+                G = G.cuda()
+
+            # Latent codes sampling
+            if args.verbose:
+                print("  \\__Sample {} {}-dimensional latent codes...".format(args.num_samples, G.dim_z))
+            zs = torch.randn(args.num_samples, G.dim_z)
+
+            if use_cuda:
+                zs = zs.cuda()
+
+            # Calculate expected latent norm
+            if args.verbose:
+                print("  \\__Calculate Jung radius...")
+            jung_radius = torch.cdist(zs, zs).max() * np.sqrt(G.dim_z / (2 * (G.dim_z + 1)))
+
+            print("jung_radius")
+            print(jung_radius)
+            print(type(jung_radius))
+
+            jung_radii_dict[gan]['Z'] = (0.0, jung_radius.cpu().detach().item())
+
+    # Save expected latent norms dictionary
+    with open(osp.join('models', 'jung_radii.json'), 'w') as fp:
+        json.dump(jung_radii_dict, fp)
+
+
+if __name__ == '__main__':
+    main()

ContraCLIP/checkpoint2model.py

@@ -0,0 +1,51 @@
+import argparse
+import os.path as osp
+import torch
+
+
+def main():
+    """An auxiliary script for converting a checkpoint file (`checkpoint.pt`) into a support sets (`support_sets.pt`)
+    and a reconstructor (`reconstructor.pt`) weights files.
+
+    Options:
+        ================================================================================================================
+        --exp : set experiment's wip model dir, as created by `train.py`, i.e., it should contain a sub-directory
+                `models/` with a checkpoint file (`checkpoint.pt`). Checkpoint file contains the weights of the
+                 support sets and the reconstructor at an intermediate stage of training (after a given iteration).
+        ================================================================================================================
+    """
+    parser = argparse.ArgumentParser(description="Convert a checkpoint file into a support sets and a reconstructor "
+                                                 "weights files")
+    parser.add_argument('--exp', type=str, required=True, help="set experiment's model dir (created by `train.py`)")
+
+    # Parse given arguments
+    args = parser.parse_args()
+
+    # Check structure of `args.exp`
+    if not osp.isdir(args.exp):
+        raise NotADirectoryError("Invalid given directory: {}".format(args.exp))
+    models_dir = osp.join(args.exp, 'models')
+    if not osp.isdir(models_dir):
+        raise NotADirectoryError("Invalid models directory: {}".format(models_dir))
+    checkpoint_file = osp.join(models_dir, 'checkpoint.pt')
+    if not osp.isfile(checkpoint_file):
+        raise FileNotFoundError("Checkpoint file not found: {}".format(checkpoint_file))
+
+    print("#. Convert checkpoint file into support sets and reconstructor weight files...")
+
+    # Load checkpoint file
+    checkpoint_dict = torch.load(checkpoint_file)
+    print("  \\__Checkpoint dictionary: {}".format(checkpoint_dict.keys()))
+
+    # Get checkpoint iteration
+    checkpoint_iter = checkpoint_dict['iter']
+    print("  \\__Checkpoint iteration: {}".format(checkpoint_iter))
+
+    # Save latent support sets (LSS) weights file
+    print("  \\__Save checkpoint latent support sets LSS weights file...")
+    torch.save(checkpoint_dict['latent_support_sets'],
+               osp.join(models_dir, 'latent_support_sets-{:07d}.pt'.format(checkpoint_iter)))
+
+
+if __name__ == '__main__':
+    main()

ContraCLIP/download_models.py

@@ -0,0 +1,168 @@
+import sys
+import os
+import os.path as osp
+import argparse
+import hashlib
+import tarfile
+import time
+import urllib.request
+from lib import GENFORCE, GENFORCE_MODELS, SFD, ARCFACE, FAIRFACE, HOPENET, AUDET, CELEBA_ATTRIBUTES, ContraCLIP_models
+
+
+def reporthook(count, block_size, total_size):
+    global start_time
+    if count == 0:
+        start_time = time.time()
+        return
+    duration = time.time() - start_time
+    progress_size = int(count * block_size)
+    speed = int(progress_size / (1024 * duration))
+    percent = min(int(count * block_size * 100 / total_size), 100)
+    sys.stdout.write("\r      \\__%d%%, %d MB, %d KB/s, %d seconds passed" %
+                     (percent, progress_size / (1024 * 1024), speed, duration))
+
+    sys.stdout.flush()
+
+
+def download(src, sha256sum, dest):
+    tmp_tar = osp.join(dest, ".tmp.tar")
+    try:
+        urllib.request.urlretrieve(src, tmp_tar, reporthook)
+    except:
+        raise ConnectionError("Error: {}".format(src))
+
+    sha256_hash = hashlib.sha256()
+    with open(tmp_tar, "rb") as f:
+        # Read and update hash string value in blocks of 4K
+        for byte_block in iter(lambda: f.read(4096), b""):
+            sha256_hash.update(byte_block)
+
+        sha256_check = sha256_hash.hexdigest() == sha256sum
+        print()
+        print("      \\__Check sha256: {}".format("OK!" if sha256_check else "Error"))
+        if not sha256_check:
+            raise Exception("Error: Invalid sha256 sum: {}".format(sha256_hash.hexdigest()))
+
+    tar_file = tarfile.open(tmp_tar, mode='r')
+    tar_file.extractall(dest)
+    os.remove(tmp_tar)
+
+
+def main():
+    """Download pre-trained GAN generators and various pre-trained detectors (used only during testing), as well as
+    pre-trained ContraCLIP models:
+    -- GenForce GAN generators [1]
+    -- SFD face detector [2]
+    -- ArcFace [3]
+    -- FairFace [4]
+    -- Hopenet [5]
+    -- AU detector [6] for 12 DISFA [7] Action Units
+    -- Facial attributes detector [8] for 5 CelebA [9] attributes
+    -- ContraCLIP [10] pre-trained models:
+        StyleGAN2@FFHQ
+        ProgGAN@CelebA-HQ:
+        StyleGAN2@AFHQ-Cats
+        StyleGAN2@AFHQ-Dogs
+        StyleGAN2@AFHQ-Cars
+
+    References:
+         [1] https://genforce.github.io/
+         [2] Zhang, Shifeng, et al. "S3FD: Single shot scale-invariant face detector." Proceedings of the IEEE
+             international conference on computer vision. 2017.
+         [3] Deng, Jiankang, et al. "Arcface: Additive angular margin loss for deep face recognition."
+             Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
+         [4] Karkkainen, Kimmo, and Jungseock Joo. "FairFace: Face attribute dataset for balanced race, gender, and age."
+             arXiv preprint arXiv:1908.04913 (2019).
+         [5] Doosti, Bardia, et al. "Hope-net: A graph-based model for hand-object pose estimation." Proceedings of the
+             IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
+         [6] Ntinou, Ioanna, et al. "A transfer learning approach to heatmap regression for action unit intensity
+             estimation." IEEE Transactions on Affective Computing (2021).
+         [7] Mavadati, S. Mohammad, et al. "DISFA: A spontaneous facial action intensity database." IEEE Transactions on
+             Affective Computing 4.2 (2013): 151-160.
+         [8] Jiang, Yuming, et al. "Talk-to-Edit: Fine-Grained Facial Editing via Dialog." Proceedings of the IEEE/CVF
+             International Conference on Computer Vision. 2021.
+         [9] Liu, Ziwei, et al. "Deep learning face attributes in the wild." Proceedings of the IEEE international
+             conference on computer vision. 2015.
+        [10] Tzelepis, C., Oldfield, J., Tzimiropoulos, G., & Patras, I. (2022). ContraCLIP: Interpretable GAN
+             generation driven by pairs of contrasting sentences. arXiv preprint arXiv:2206.02104.
+    """
+    parser = argparse.ArgumentParser(description="Download pre-trained models")
+    parser.add_argument('-m', '--contraclip-models', action='store_true', help="download pre-trained ContraCLIP models")
+    args = parser.parse_args()
+
+    # Create pre-trained models root directory
+    pretrained_models_root = osp.join('models', 'pretrained')
+    os.makedirs(pretrained_models_root, exist_ok=True)
+
+    # Download the following pre-trained GAN generators (under models/pretrained/)
+    print("#. Download pre-trained GAN generators...")
+    print("  \\__.GenForce")
+    download_genforce_models = False
+    for k, v in GENFORCE_MODELS.items():
+        if not osp.exists(osp.join(pretrained_models_root, 'genforce', v[0])):
+            download_genforce_models = True
+            break
+    if download_genforce_models:
+        download(src=GENFORCE[0], sha256sum=GENFORCE[1], dest=pretrained_models_root)
+    else:
+        print("      \\__Already exists.")
+
+    print("#. Download pre-trained ArcFace model...")
+    print("  \\__.ArcFace")
+    if osp.exists(osp.join(pretrained_models_root, 'arcface', 'model_ir_se50.pth')):
+        print("      \\__Already exists.")
+    else:
+        download(src=ARCFACE[0], sha256sum=ARCFACE[1], dest=pretrained_models_root)
+
+    print("#. Download pre-trained SFD face detector model...")
+    print("  \\__.Face detector (SFD)")
+    if osp.exists(osp.join(pretrained_models_root, 'sfd', 's3fd-619a316812.pth')):
+        print("      \\__Already exists.")
+    else:
+        download(src=SFD[0], sha256sum=SFD[1], dest=pretrained_models_root)
+
+    print("#. Download pre-trained FairFace model...")
+    print("  \\__.FairFace")
+    if osp.exists(osp.join(pretrained_models_root, 'fairface', 'fairface_alldata_4race_20191111.pt')) and \
+            osp.exists(osp.join(pretrained_models_root, 'fairface', 'res34_fair_align_multi_7_20190809.pt')):
+        print("      \\__Already exists.")
+    else:
+        download(src=FAIRFACE[0], sha256sum=FAIRFACE[1], dest=pretrained_models_root)
+
+    print("#. Download pre-trained Hopenet model...")
+    print("  \\__.Hopenet")
+    if osp.exists(osp.join(pretrained_models_root, 'hopenet', 'hopenet_alpha1.pkl')) and \
+            osp.exists(osp.join(pretrained_models_root, 'hopenet', 'hopenet_alpha2.pkl')) and \
+            osp.exists(osp.join(pretrained_models_root, 'hopenet', 'hopenet_robust_alpha1.pkl')):
+        print("      \\__Already exists.")
+    else:
+        download(src=HOPENET[0], sha256sum=HOPENET[1], dest=pretrained_models_root)
+
+    print("#. Download pre-trained AU detector model...")
+    print("  \\__.FANet")
+    if osp.exists(osp.join(pretrained_models_root, 'au_detector', 'disfa_adaptation_f0.pth')):
+        print("      \\__Already exists.")
+    else:
+        download(src=AUDET[0], sha256sum=AUDET[1], dest=pretrained_models_root)
+
+    print("#. Download pre-trained CelebA attributes predictors models...")
+    print("  \\__.CelebA")
+    if osp.exists(osp.join(pretrained_models_root, 'celeba_attributes', 'eval_predictor.pth.tar')):
+        print("      \\__Already exists.")
+    else:
+        download(src=CELEBA_ATTRIBUTES[0], sha256sum=CELEBA_ATTRIBUTES[1], dest=pretrained_models_root)
+
+    # Download pre-trained ContraCLIP models
+    if args.contraclip_models:
+        pretrained_contraclip_root = osp.join('experiments', 'complete')
+        os.makedirs(pretrained_contraclip_root, exist_ok=True)
+
+        print("#. Download pre-trained ContraCLIP models...")
+        print("  \\__.ContraCLIP pre-trained models...")
+        download(src=ContraCLIP_models[0],
+                 sha256sum=ContraCLIP_models[1],
+                 dest=pretrained_contraclip_root)
+
+
+if __name__ == '__main__':
+    main()

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/02be4f3503db069a28be3bf222c0f64ae6f85d05/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ce54ca0a139e42f1c79fe7f60d576d4a485e36627318c7c246275dee69a15ee
+size 2795

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/10c29d1257e7c6e513d8ef23599ba6ba89eda181/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a511c6edd052939a88acc05a299f3da41a5b1f05270d2443fd8a8e916bd05f1
+size 2795

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/323234c425e1b4fd5ec0539bb64765d72afffc75/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70da19d64562dae4c03e15617a55024c30070f7419ed9d32adcfe5d5240b7adb
+size 2795

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/555510a5999a3c5eb3097e0b80da4cee97088c8e/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8342234862b5c57bcf3c4153837f6654052f1de4192f1e7d0a464a9fc0360550
+size 2795

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/911ea1a1d3b3e6b57a819ad9310048384608ce08/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a5e51d71a466329e2396c8124758dfe3d05f4bab42c2d782b4e4f77af30bccb
+size 2795

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/9232b69c406fece5016ccfe260a226eaef1d9181/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e6f5a35eb58c4c2e7228f423e5860730cdb0d212855fc374fc471702d6b3339
+size 2795

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/c1345dab91e4c82070858e3201bcd7eac0bb042e/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5afd158af814aaf1aae0455127b3d5eca7d1e599cb9bad2f45f1d1a7eb4fbee7
+size 2795

ContraCLIP/experiments/latent_codes/pggan_celebahq1024/pggan_celebahq1024-8/ce26bd5026197c1df60bc43ab1a99f3db8730b0a/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfaa6a1a2189cbdce8d8371428d259aaa9de83fa3975352be57694ca3e3c1144
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/08307a8eacf4509f45ab65e8ee76dc53d089dec9/latent_code_w+.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82712b3f877423756238c59c19cdc49b131aeab9054e977e0eabce7dae881261
+size 33515

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/08307a8eacf4509f45ab65e8ee76dc53d089dec9/latent_code_w.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d3f23f63d0002b4bd330fcabad40161443181b380c2ee343f381d6f57406410
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/08307a8eacf4509f45ab65e8ee76dc53d089dec9/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cee26d0ce68a05d32fcb28eeb8a13418c1978c62c3f3daab5b97fe48fb6a8cc8
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/28c1c29df1be16a26914078f57b2b95598496048/latent_code_w+.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff0a3002d8d50bbc7702c1390726fddb57fa1d384e6b3caecdb462db0aa272a2
+size 33515

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/28c1c29df1be16a26914078f57b2b95598496048/latent_code_w.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c19769cbebb4bf0278d55ebc46442a54d56241487fdf26eaf1cd7e392959809a
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/28c1c29df1be16a26914078f57b2b95598496048/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:562f242b6885c9a829a8f4b0e6af453f3dd53684aa182a4ef6b90d871cada48f
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/3ac589d77dc2845eda68b3e92b92f5aef972bd93/latent_code_w+.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d745141e50c6a3e69ecd5480ad9f22547e78854c2024b3f43ce9b4bcc5627565
+size 33515

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/3ac589d77dc2845eda68b3e92b92f5aef972bd93/latent_code_w.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4d787702e6e28f7a54afccbaa2c26bbd728af5c1edb33ee15f5c214094590a7
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/3ac589d77dc2845eda68b3e92b92f5aef972bd93/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:874978425c2472ec203918a10d359dd1e9cd9c795939de4263da071a31bc68d2
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/565fd0382c69e4c9462179dbce46cab36b576226/latent_code_w+.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccfe3e29ba646c227f286e2554b758e17f339562ef093e6a66e8a86bb8a70948
+size 33515

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/565fd0382c69e4c9462179dbce46cab36b576226/latent_code_w.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e3b76d1580404c16254926af0567e1a14478a4343c30f47d2666f5b3ab2cb80
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/565fd0382c69e4c9462179dbce46cab36b576226/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bee2b9d54714f6a7b8ac46a709599cfdffe20b550f0301d1069d8264855d6889
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/584c090fdba130d896e7b67f942df55f44baf022/latent_code_w+.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66213a5a461870c4b818be52684f06475076904d3458a487aadb3d49bd9427b1
+size 33515

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/584c090fdba130d896e7b67f942df55f44baf022/latent_code_w.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:001cae0eb200dec5984724bb15f5b55abc69d2f76e6ff9ada0628d2c56fa7b80
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/584c090fdba130d896e7b67f942df55f44baf022/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89eddb1448b8b7cab265f23114248dc6ef3f9b398b188ed4237c1488d1d31b4c
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/60ace58591602b942ef7816000203c07479baf1e/latent_code_w+.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bed42b35a7ed1da7fa51000244723e23b662b1873f68403ec94b3098c79e696
+size 33515

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/60ace58591602b942ef7816000203c07479baf1e/latent_code_w.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cce019f116bb2d7e1999b6a160e0cfb405712953616433a64ff076ddf387997
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/60ace58591602b942ef7816000203c07479baf1e/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c464e73f08f4b5288848c1ee775b36d79c45ef1533aec4ef99f09c8b4a125c2
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/6e3a4bd20238f6964cb447efc2bf4f9ae889212f/latent_code_w+.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3aaa266547d4a8ea6e40c399c8e2923c5b7c5b86ec463ba3e6ec3f47103fa28d
+size 33515

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/6e3a4bd20238f6964cb447efc2bf4f9ae889212f/latent_code_w.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2107d9d5e9e2ae036dc8ec731b7e335d3ab9f0a58918086101804a3197280210
+size 2795

ContraCLIP/experiments/latent_codes/stylegan2_afhqcat512/stylegan2_afhqcat512-16/6e3a4bd20238f6964cb447efc2bf4f9ae889212f/latent_code_z.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:010c319d0f4638ef021dd33a819c17c3f9fa8c5bf4403d2c2c462c335d790d54
+size 2795