|
---
|
|
license: apache-2.0
|
|
---
|
|
|
|
# ELC-ParserBERT
|
|
|
|
This model is an adaptation of the [Every Layer Counts BERT model](<https://aclanthology.org/2023.conll-babylm.20/>), but it incorporates the `Parser Network` from the [StructFormer](<https://arxiv.org/abs/2012.00857>). It was trained for the [BabyLM 2024 challenge](https://babylm.github.io/index.html)'s Strict-Small track.
|
|
|
|
## Dataset
|
|
|
|
The training data for the challenge can be accessed through OSF [here](https://osf.io/ad7qg/). This model was trained on the 10M token training dataset.
|
|
|
|
### Order in Pretraining
|
|
|
|
After the segmentation of the data, the segements are ordered in increasing difficulty according to the flesch_reading_ease metric. This ordering can either be maintained by not including the shuffle flag when training or rejected (and allowing shuffling of the data to happen); this model did shuffle the data.
|
|
|
|
## Hyperparameters
|
|
|
|
### Base Model
|
|
|
|
| Hyperparameter | Value |
|
|
| -------------- | ----- |
|
|
| Initial learning rate | 5e-3 |
|
|
| Batch size | 256 |
|
|
| Steps | 13495 |
|
|
| shuffled | True |
|
|
|attention_probs_dropout_prob | 0.1 |
|
|
| classifier_dropout | 0.2 |
|
|
| hidden_dropout_prob | 0.1 |
|
|
| hidden_size | 384 |
|
|
| intermediate_size | 1024 |
|
|
| layer_norm_eps | 1e-07 |
|
|
| max_position_embeddings | 512 |
|
|
| num_attention_heads | 6 |
|
|
| num_hidden_layers | 12 |
|
|
| vocab_size | 16384 |
|
|
| n_parser_layers | 4 |
|
|
| parser_conv_size |9 |
|
|
|
|
### Fine-tuning
|
|
|
|
The fine-tuning parameters were unchanged from the organizer outside of following the ELC-BERT model's patience approach for last year, in particular:
|
|
|
|
| Hyperparameter | Value |
|
|
| -------------- | ----- |
|
|
| Initial learning rate | 5e-5 |
|
|
| Batch size | 64 |
|
|
| Maximum epochs | 10 |
|
|
| Evaluate every (epochs) | 1 |
|
|
| Patience | 10 (for CoLA, MRPC, RTE, BoolQ, MultiRC, and WSC), 100 (for MNLI, MNLI-MM, QQP, QNLI, and SST-2) |
|
|
| Seed | 12 |
|
|
|
|
## Credit
|
|
|
|
As mentioned above, this model is an adapatation of Every Layer Counts (ELC) BERT and StructFormer, the citations and code repositories for which can be found here
|
|
|
|
* StructFormer
|
|
* [StructFormer Github](<https://github.com/google-research/google-research/tree/master/structformer>)
|
|
|
|
* ```bibtex
|
|
@misc{shen2020structformer,
|
|
title={StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling},
|
|
author={Yikang Shen and Yi Tay and Che Zheng and Dara Bahri and Donald Metzler and Aaron Courville},
|
|
year={2020},
|
|
eprint={2012.00857},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL}}```
|
|
* ELC-BERT:
|
|
* [ELC-BERT Github](<https://github.com/ltgoslo/elc-bert>)
|
|
* [ELC-BERT 10M Hugging Face](https://huggingface.co/lgcharpe/ELC_BERT_small_baby_10M)
|
|
* ```bibtex
|
|
@inproceedings{georges-gabriel-charpentier-samuel-2023-layers,
|
|
title = "Not all layers are equally as important: Every Layer Counts {BERT}",
|
|
author = "Georges Gabriel Charpentier, Lucas and
|
|
Samuel, David",
|
|
editor = "Warstadt, Alex and
|
|
Mueller, Aaron and
|
|
Choshen, Leshem and
|
|
Wilcox, Ethan and
|
|
Zhuang, Chengxu and
|
|
Ciro, Juan and
|
|
Mosquera, Rafael and
|
|
Paranjabe, Bhargavi and
|
|
Williams, Adina and
|
|
Linzen, Tal and
|
|
Cotterell, Ryan",
|
|
booktitle = "Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning",
|
|
month = dec,
|
|
year = "2023",
|
|
address = "Singapore",
|
|
publisher = "Association for Computational Linguistics",
|
|
url = "https://aclanthology.org/2023.conll-babylm.20",
|
|
doi = "10.18653/v1/2023.conll-babylm.20",
|
|
pages = "238--252",
|
|
}```
|
|
|
