---
license: apache-2.0
tags:
- distigpt2
- hearthstone
metrics:
- bleu
- dvitel/codebleu
- exact_match
- chrf
datasets:
- dvitel/hearthstone
model-index:
- name: h0
  results:
  - task:
      type: text-generation
      name: Python Code Synthesis
    dataset:
      type: dvitel/hearthstone
      name: HearthStone
      split: test
    metrics:
      - type: exact_match
        value: 0.30303030303030304
        name: Exact Match
      - type: bleu
        value: 0.8850182403024257
        name: BLEU        
      - type: dvitel/codebleu
        value: 0.677852377992836
        name: CodeBLEU                
      - type: chrf
        value: 91.00848749530383
        name: chrF     
---

# h3

This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on [hearthstone](https://huggingface.co/datasets/dvitel/hearthstone) dataset.
[GitHub repo](https://github.com/dvitel/nlp-sem-parsing/blob/master/h3.py).
It achieves the following results on the evaluation set:
- Loss: 0.2782
- Exact Match: 0.2879
- Bleu: 0.9121
- Codebleu: 0.7482
- Ngram Match Score: 0.7504
- Weighted Ngram Match Score: 0.7583
- Syntax Match Score: 0.7673
- Dataflow Match Score: 0.7169
- Chrf: 93.1064

## Model description

DistilGPT2 fine-tuned on HearthStone dataset for 200 epochs. \
Related to [dvitel/h0](https://huggingface.co/dvitel/h0) but with preprocessing which anonymizes classes and function variables (Local renaming). \
[dvitel/h2](https://huggingface.co/dvitel/h2) implements global renaming where all names are removed. Global renaming showed worse results compared to local renaming.

## Intended uses & limitations

HearthStone card code synthesis. 

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 17
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 200
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step  | Validation Loss | Exact Match | Bleu   | Codebleu | Ngram Match Score | Weighted Ngram Match Score | Syntax Match Score | Dataflow Match Score | Chrf    |
|:-------------:|:------:|:-----:|:---------------:|:-----------:|:------:|:--------:|:-----------------:|:--------------------------:|:------------------:|:--------------------:|:-------:|
| 0.8612        | 11.94  | 1600  | 0.2725          | 0.0455      | 0.8477 | 0.6050   | 0.6229            | 0.6335                     | 0.6203             | 0.5431               | 88.7010 |
| 0.175         | 23.88  | 3200  | 0.2311          | 0.0909      | 0.8739 | 0.6304   | 0.6566            | 0.6656                     | 0.6484             | 0.5508               | 90.7364 |
| 0.1036        | 35.82  | 4800  | 0.2172          | 0.1818      | 0.8930 | 0.6905   | 0.6976            | 0.7062                     | 0.7172             | 0.6409               | 91.9702 |
| 0.0695        | 47.76  | 6400  | 0.2233          | 0.2424      | 0.8944 | 0.7017   | 0.7148            | 0.7232                     | 0.7187             | 0.6499               | 92.0340 |
| 0.0482        | 59.7   | 8000  | 0.2407          | 0.2879      | 0.9046 | 0.7301   | 0.7387            | 0.7456                     | 0.7475             | 0.6885               | 92.6219 |
| 0.0352        | 71.64  | 9600  | 0.2407          | 0.2424      | 0.9074 | 0.7255   | 0.7371            | 0.7448                     | 0.7482             | 0.6718               | 92.8281 |
| 0.0262        | 83.58  | 11200 | 0.2596          | 0.3030      | 0.9061 | 0.7445   | 0.7415            | 0.7500                     | 0.7774             | 0.7091               | 92.6737 |
| 0.0213        | 95.52  | 12800 | 0.2589          | 0.2879      | 0.9061 | 0.7308   | 0.7409            | 0.7488                     | 0.7464             | 0.6873               | 92.7814 |
| 0.0164        | 107.46 | 14400 | 0.2679          | 0.2879      | 0.9096 | 0.7452   | 0.7510            | 0.7592                     | 0.7626             | 0.7079               | 92.9900 |
| 0.0131        | 119.4  | 16000 | 0.2660          | 0.2879      | 0.9096 | 0.7447   | 0.7480            | 0.7564                     | 0.7666             | 0.7079               | 93.0122 |
| 0.0116        | 131.34 | 17600 | 0.2669          | 0.2727      | 0.9092 | 0.7463   | 0.7445            | 0.7529                     | 0.7684             | 0.7194               | 92.9256 |
| 0.0093        | 143.28 | 19200 | 0.2678          | 0.2879      | 0.9113 | 0.7531   | 0.7496            | 0.7581                     | 0.7709             | 0.7336               | 93.0406 |
| 0.0083        | 155.22 | 20800 | 0.2728          | 0.2879      | 0.9103 | 0.7407   | 0.7462            | 0.7540                     | 0.7702             | 0.6924               | 92.9302 |
| 0.0077        | 167.16 | 22400 | 0.2774          | 0.2879      | 0.9103 | 0.7449   | 0.7449            | 0.7532                     | 0.7659             | 0.7156               | 92.9742 |
| 0.0069        | 179.1  | 24000 | 0.2774          | 0.2879      | 0.9120 | 0.7396   | 0.7463            | 0.7539                     | 0.7633             | 0.6950               | 93.1057 |
| 0.0069        | 191.04 | 25600 | 0.2782          | 0.2879      | 0.9121 | 0.7482   | 0.7504            | 0.7583                     | 0.7673             | 0.7169               | 93.1064 |


### Framework versions

- Transformers 4.24.0
- Pytorch 1.13.0
- Datasets 2.6.1
- Tokenizers 0.13.1