# Knowledge distillation: A good teacher is patient and consistent
*by Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, Alexander Kolesnikov*

## Introduction
We publish all teacher models, and configurations for the main experiments of
the paper, as well as training logs and student models.

Please read the main [big_vision README](/README.md) to learn how to run
configs, and remember that each config file contains an example invocation in
the top-level comment.

## Results

We provide the following [colab to read and plot the logfiles](https://colab.research.google.com/drive/1nMykzUzsfQ_uAxfj3k35DYsATnG_knPl?usp=sharing)
of a few runs that we reproduced on Cloud.

### ImageNet-1k

The file [bit_i1k.py](bit_i1k.py) is the configuration which reproduces our
distillation runs on ImageNet-1k reported in Figures 1 and 5(left) and the first
row of Table1.

We release both student and teacher models:

| Model      | Download link | Resolution  | ImageNet top-1 acc. (paper) | 
| :---       | :---:         | :---:       |  :---:                      |
| BiT-R50x1  | [link](https://storage.googleapis.com/bit_models/distill/R50x1_160.npz)      | 160 |  80.5 |
| BiT-R50x1  | [link](https://storage.googleapis.com/bit_models/distill/R50x1_224.npz)      | 224 |  82.8 |
| BiT-R152x2 | [link](https://storage.googleapis.com/bit_models/distill/R152x2_T_224.npz)   | 224 |  83.0 |
| BiT-R152x2 | [link](https://storage.googleapis.com/bit_models/distill/R152x2_T_384.npz)   | 384 |  84.3 |

### Flowers/Pet/Food/Sun

The files [bigsweep_flowers_pet.py](bigsweep_flowers_pet.py) and
[bigsweep_food_sun.py](bigsweep_food_sun.py) can be used to reproduce the
distillation runs on these datasets and shown in Figures 3,4,9-12, and Table4.

While our open-source release does not currently support doing hyper-parameter
sweeps, we still provide an example of the sweeps at the end of the configs
for reference.

### Teacher models
Links to all teacher models we used can be found in [common.py](common.py).