|
--- |
|
language: fr |
|
license: mit |
|
tags: |
|
- causal-lm |
|
- fr |
|
datasets: |
|
- c4 |
|
- The Pile |
|
--- |
|
|
|
### Quantized Cedille/fr-boris with 8-bit weights |
|
|
|
|
|
This is a version of Cedille's GPT-J (fr-boris) with 6 billion parameters that is modified so you can generate **and fine-tune the model in colab or equivalent desktop gpu (e.g. single 1080Ti)**. Inspired by [GPT-J 8bit](https://huggingface.co/hivemind/gpt-j-6B-8bit). |
|
|
|
Here's how to run it: [](https://colab.research.google.com/drive/1ft6wQU0BhqG5PRlwgaZJv2VukKKjU4Es) |
|
|
|
This model can be easily loaded using the `GPTJForCausalLM` functionality: |
|
```python |
|
from transformers import GPTJForCausalLM |
|
model = GPTJForCausalLM.from_pretrained("gustavecortal/fr-boris-8bit") |
|
``` |
|
|
|
## fr-boris |
|
|
|
Boris is a 6B parameter autoregressive language model based on the GPT-J architecture and trained using the [mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) codebase. |
|
|
|
Boris was trained on around 78B tokens of French text from the [C4](https://huggingface.co/datasets/c4) dataset. |
|
|
|
## Links |
|
|
|
* [Cedille](https://en.cedille.ai/) |
|
* [Hivemind](https://training-transformers-together.github.io/) |
|
* [Gustave Cortal](https://twitter.com/gustavecortal) |
