Compact Convolutional Transformers

Based on the Compact Convolutional Transformers example on keras.io created by Sayak Paul.

Model description

As discussed in the Vision Transformers (ViT) paper, a Transformer-based architecture for vision typically requires a larger dataset than usual, as well as a longer pre-training schedule. ImageNet-1k (which has about a million images) is considered to fall under the medium-sized data regime with respect to ViTs. This is primarily because, unlike CNNs, ViTs (or a typical Transformer-based architecture) do not have well-informed inductive biases (such as convolutions for processing images). This begs the question: can't we combine the benefits of convolution and the benefits of Transformers in a single network architecture? These benefits include parameter-efficiency, and self-attention to process long-range and global dependencies (interactions between different regions in an image).

In Escaping the Big Data Paradigm with Compact Transformers, Hassani et al. present an approach for doing exactly this. They proposed the Compact Convolutional Transformer (CCT) architecture.

Intended uses & limitations

In the original paper, the authors use AutoAugment to induce stronger regularization. In this example, the standard geometric augmentations (like random cropping and flipping) are used.
The CCT model was trained for 30 epochs. Its plot in the 'Training Metrics' tab shows no signs of overfitting. This means that this network can be trained for longer (perhaps with a bit more regularization) and better performance may be obtained. This performance can further be improved by additional recipes like cosine decay learning rate schedule, other data augmentation techniques like AutoAugment, MixUp or Cutmix.

Training and evaluation data

The model is trained using the CIFAR-10 dataset.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

name	learning_rate	decay	beta_1	beta_2	epsilon	amsgrad	weight_decay	exclude_from_weight_decay	training_precision
AdamW	0.0010000000474974513	0.0	0.8999999761581421	0.9990000128746033	1e-07	False	9.999999747378752e-05	None	float32

Model Plot

View Model Plot

Model reproduced by Edoardo Abati

keras-io
/

cct