Create README.md

README.md

@@ -0,0 +1,61 @@
+---
+license: apache-2.0
+---
+license: apache-2.0
+---
+
+**Paper**: [Adapting Language Models to Compress Contexts](https://arxiv.org/abs/2305.14788)  
+
+**Code**: https://github.com/princeton-nlp/AutoCompressors
+
+**Models**: 
+- Llama-2-7b fine-tuned models: [AutoCompressor-Llama-2-7b-6k](https://huggingface.co/princeton-nlp/AutoCompressor-Llama-2-7b-6k/), [FullAttention-Llama-2-7b-6k](https://huggingface.co/princeton-nlp/FullAttention-Llama-2-7b-6k)
+- OPT-2.7b fine-tuned models:  [AutoCompressor-2.7b-6k](https://huggingface.co/princeton-nlp/AutoCompressor-2.7b-6k), [AutoCompressor-2.7b-30k](https://huggingface.co/princeton-nlp/AutoCompressor-2.7b-30k), [RMT-2.7b-8k](https://huggingface.co/princeton-nlp/RMT-2.7b-8k), [FullAttention-2.7b-4k](https://huggingface.co/princeton-nlp/FullAttention-2.7b-4k)
+- OPT-1.3b fine-tuned models: [AutoCompressor-1.3b-30k](https://huggingface.co/princeton-nlp/AutoCompressor-1.3b-30k), [RMT-1.3b-30k](https://huggingface.co/princeton-nlp/RMT-1.3b-30k)
+
+---
+
+FullAttention-2.7b-4k is a model fine-tuned from [facebook/opt-2.7b](https://huggingface.co/facebook/opt-2.7b) following the context window extension method described in [Adapting Language Models to Compress Contexts](https://arxiv.org/abs/2305.14788). 
+The 2,048 positional embeddings of the pre-trained OPT-2.7b are duplicated and the model is fine-tuned on sequences of 4,096 tokens from 2B tokens from  [The Pile](https://pile.eleuther.ai). 
+
+To get started, download the [`AutoCompressor`](https://github.com/princeton-nlp/AutoCompressors) repository and load the model as follows:
+
+```
+from auto_compressor import AutoCompressorModel
+
+model = AutoCompressorModel.from_pretrained("princeton-nlp/FullAttention-2.7b-4k")
+``` 
+
+**Evaluation**
+
+We record the perplexity achieved by our OPT-2.7b models on segments of 2,048 tokens, conditioned on different amounts of context. 
+FullAttention-2.7-4k  uses full uncompressed contexts whereas AutoCompressor-2.7b-6k and RMT-2.7b-8k compress segments of 2,048 tokens into 50 summary vectors.
+
+*In-domain Evaluation*
+
+| Context Tokens               | 0   |512  | 2048 | 4096 | 6144 |
+| -----------------------------|-----|-----|------|------|------|
+| FullAttention-2.7b-4k        | 6.57|6.15 |5.94  |-     |-     |
+| RMT-2.7b-8k                  | 6.34|6.19 |6.02 | 6.02  | 6.01 |
+| AutoCompressor-2.7b-6k       | 6.31|6.04 | 5.98 | 5.94 | 5.93 | 
+
+*Out-of-domain Evaluation*
+
+| Context Tokens               | 0   |512  | 2048 | 4096 | 6144 |
+| -----------------------------|-----|-----|------|------|------|
+| FullAttention-2.7b-4k        | 8.94|8.28 |7.93  |-     |-     |
+| RMT-2.7b-8k                  | 8.62|8.44 |8.21 | 8.20  | 8.20 |
+| AutoCompressor-2.7b-6k       | 8.60|8.26 | 8.17 | 8.12 | 8.10 | 
+
+
+## Bibtex
+```
+@misc{chevalier2023adapting,
+      title={Adapting Language Models to Compress Contexts}, 
+      author={Alexis Chevalier and Alexander Wettig and Anirudh Ajith and Danqi Chen},
+      year={2023},
+      eprint={2305.14788},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```