Update README.md
Browse files
README.md
CHANGED
@@ -12,7 +12,7 @@ language:
|
|
12 |
|
13 |
# ipt-350m
|
14 |
|
15 |
-
ipt-350m is a decoder-style transformer pretrained from scratch on ~13B tokens of Italian text.
|
16 |
|
17 |
It uses a modified transformer architecture optimized for efficient training and inference. Positional embeddings are replaced with Attention with Linear Biases ([ALiBi](https://arxiv.org/abs/2108.12409)).
|
18 |
|
|
|
12 |
|
13 |
# ipt-350m
|
14 |
|
15 |
+
ipt-350m is a decoder-style transformer pretrained from scratch on ~13B tokens of Italian text (wip: trained on unfiltered oscar).
|
16 |
|
17 |
It uses a modified transformer architecture optimized for efficient training and inference. Positional embeddings are replaced with Attention with Linear Biases ([ALiBi](https://arxiv.org/abs/2108.12409)).
|
18 |
|