# Results 

The results here are taken from running `score_predictions.py` from the [babylm evaluation pipeline](https://github.com/babylm/evaluation-pipeline-2024) on the `ELC_ParserBERT_10M_textonly_predictions.json.gz` file in this directory, which contains the predictions for the different evaluation tasks.

## Overall Results

Here are the average results per section and the macroscore, compared with the baseline models:

| Model | BLiMP | BLiMP Supplement | EWoK | GLUE | *Macroaverage* |
| --- | --- | --- | --- | --- | --- |
| BabyLlama | 69.8 | 59.5 | 50.7 | 63.3 | 60.8 |
| LTG-BERT | 60.6 | 60.8 | 48.9 | 60.3 | 57.7 |
| ELC-ParserBERT | 59.6 | 57.7 | 63.1 | 44.5 | 56.2 |

## The Breakdown Per Section

|glue subtask   |    Score | 
|-------------- |  ------- |
|cola (MCC)     |    0.042 |
|sst2           |    0.502 |
|mrpc (F1)      |    0.82  |
|qqp (F1)       |    0     |
|mnli           |    0.357 |
|mnli-mm        |    0.355 |
|qnli           |    0.491 |
|rte            |    0.496 |
|boolq          |    0.585 |
|multirc        |    0.63  |
|wsc            |    0.615 |
|*Average*      |    0.445 |

| blimp subtask                                       |   Score  |
| --------------------------------------------------- | -------  |
| adjunct_island                                      |   0.712  |
| anaphor_gender_agreement                            |   0.593  |
| anaphor_number_agreement                            |   0.647  |
| animate_subject_passive                             |   0.594  |
| animate_subject_trans                               |   0.47   |
| causative                                           |   0.726  |
| complex_NP_island                                   |   0.447  |
| coordinate_structure_constraint_complex_left_branch |   0.39   |
| coordinate_structure_constraint_object_extraction   |   0.806  |
| determiner_noun_agreement_1                         |   0.793  |
| determiner_noun_agreement_2                         |   0.936  |
| determiner_noun_agreement_irregular_1               |   0.467  |
| determiner_noun_agreement_irregular_2               |   0.394  |
| determiner_noun_agreement_with_adj_2                |   0.889  |
| determiner_noun_agreement_with_adj_irregular_1      |   0.834  |
| determiner_noun_agreement_with_adj_irregular_2      |   0.848  |
| determiner_noun_agreement_with_adjective_1          |   0.758  |
| distractor_agreement_relational_noun                |   0.212  |
| distractor_agreement_relative_clause                |   0.282  |
| drop_argument                                       |   0.485  |
| ellipsis_n_bar_1                                    |   0.505  |
| ellipsis_n_bar_2                                    |   0.342  |
| existential_there_object_raising                    |   0.447  |
| existential_there_quantifiers_1                     |   0.385  |
| existential_there_quantifiers_2                     |   0.396  |
| existential_there_subject_raising                   |   0.476  |
| expletive_it_object_raising                         |   0.44   |
| inchoative                                          |   0.527  |
| intransitive                                        |   0.484  |
| irregular_past_participle_adjectives                |   0.348  |
| irregular_past_participle_verbs                     |   0.594  |
| irregular_plural_subject_verb_agreement_1           |   0.634  |
| irregular_plural_subject_verb_agreement_2           |   0.687  |
| left_branch_island_echo_question                    |   0.634  |
| left_branch_island_simple_question                  |   0.615  |
| matrix_question_npi_licensor_present                |   0.206  |
| npi_present_1                                       |   0.362  |
| npi_present_2                                       |   0.347  |
| only_npi_licensor_present                           |   0.964  |
| only_npi_scope                                      |   0.89   |
| passive_1                                           |   0.514  |
| passive_2                                           |   0.482  |
| principle_A_c_command                               |   0.635  |
| principle_A_case_1                                  |   0.999  |
| principle_A_case_2                                  |   0.78   |
| principle_A_domain_1                                |   0.893  |
| principle_A_domain_2                                |   0.623  |
| principle_A_domain_3                                |   0.556  |
| principle_A_reconstruction                          |   0.339  |
| regular_plural_subject_verb_agreement_1             |   0.628  |
| regular_plural_subject_verb_agreement_2             |   0.663  |
| sentential_negation_npi_licensor_present            |   0.93   |
| sentential_negation_npi_scope                       |   0.722  |
| sentential_subject_island                           |   0.361  |
| superlative_quantifiers_1                           |   0.702  |
| superlative_quantifiers_2                           |   0.498  |
| tough_vs_raising_1                                  |   0.351  |
| tough_vs_raising_2                                  |   0.648  |
| transitive                                          |   0.645  |
| wh_island                                           |   0.719  |
| wh_questions_object_gap                             |   0.657  |
| wh_questions_subject_gap                            |   0.861  |
| wh_questions_subject_gap_long_distance              |   0.937  |
| wh_vs_that_no_gap                                   |   0.969  |
| wh_vs_that_no_gap_long_distance                     |   0.969  |
| wh_vs_that_with_gap                                 |   0.222  |
| wh_vs_that_with_gap_long_distance                   |   0.063  |
| *Average*                                           |   0.596  |

| blimp_supplement subtask   |    Score |
| -------------------------- |  ------- |
| hypernym                   |    0.531 |
| qa_congruence_easy         |    0.641 |
| qa_congruence_tricky       |    0.521 |
| subject_aux_inversion      |    0.614 |
| turn_taking                |    0.579 |
| *Average*                  |    0.577 |

| ewok subtask            |   Score |
| ----------------------- | ------- |
| agent-properties        |   0.738 |
| material-dynamics       |   0.81  |
| material-properties     |   0.6   |
| physical-dynamics       |   0.383 |
| physical-interactions   |   0.599 |
| physical-relations      |   0.817 |
| quantitative-properties |   0.427 |
| social-interactions     |   0.565 |
| social-properties       |   0.561 |
| social-relations        |   0.807 |
| spatial-relations       |   0.635 |
| *Average*               |   0.631 |