# CLIP Transcoder Checkpoint

This model is a Transcoder trained on CLIP's internal representations.

## Model Details

### Architecture
- **Layer**: 8
- **Layer Type**: ln2.hook_normalized
- **Model**: laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
- **Dictionary Size**: 49152
- **Input Dimension**: 768
- **Expansion Factor**: 64
- **CLS Token Only**: False

### Training
- **Learning Rate**: 0.0005932154848461442
- **Batch Size**: 4096
- **Context Size**: 50

### Sparsity
- **L0 (Active Features)**: 1024


## Additional Information
- **Wandb Run**: https://wandb.ai/perceptual-alignment/openclip-transcoders/runs/zh2kpkp6/
- **Random Seed**: 42