File size: 8,462 Bytes
de27158 999b43e de27158 999b43e de27158 999b43e de27158 999b43e de27158 999b43e de27158 4153976 de27158 999b43e de27158 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 |
---
license: apache-2.0
tags:
- distigpt2
- hearthstone
metrics:
- bleu
- dvitel/codebleu
- exact_match
- chrf
datasets:
- dvitel/hearthstone
model-index:
- name: h0
results:
- task:
type: text-generation
name: Python Code Synthesis
dataset:
type: dvitel/hearthstone
name: HearthStone
split: test
metrics:
- type: exact_match
value: 0.30303030303030304
name: Exact Match
- type: bleu
value: 0.8850182403024257
name: BLEU
- type: dvitel/codebleu
value: 0.677852377992836
name: CodeBLEU
- type: chrf
value: 91.00848749530383
name: chrF
---
# h3
This model is a fine-tuned version of [distilgpt2](https://huggingface.co/distilgpt2) on [hearthstone](https://huggingface.co/datasets/dvitel/hearthstone) dataset.
[GitHub repo](https://github.com/dvitel/nlp-sem-parsing/blob/master/h3.py).
It achieves the following results on the evaluation set:
- Loss: 0.2782
- Exact Match: 0.2879
- Bleu: 0.9121
- Codebleu: 0.7482
- Ngram Match Score: 0.7504
- Weighted Ngram Match Score: 0.7583
- Syntax Match Score: 0.7673
- Dataflow Match Score: 0.7169
- Chrf: 93.1064
## Model description
DistilGPT2 fine-tuned on HearthStone dataset for 200 epochs. \
Related to [dvitel/h0](https://huggingface.co/dvitel/h0) but with preprocessing which anonymizes classes and function variables (Local renaming). \
[dvitel/h2](https://huggingface.co/dvitel/h2) implements global renaming where all names are removed. Global renaming showed worse results compared to local renaming.
Example of generated code with mistake on last eval iteration (EV L - gold labels, EV P - prediction):
```python
EV L class CLS0(MinionCard):
def __init__(self):
super().__init__('Darkscale Healer', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Heal(2), CharacterSelector()))
def create_minion(self, v0):
return Minion(4, 5)
EV P class CLS0(MinionCard):
def __init__(self):
super().__init__('Darkscale Healer', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Heal(2), CharacterSelector())
def create_minion(self, v0):
return Minion(4, 5)
EV L class CLS0(WeaponCard):
def __init__(self):
super().__init__('Fiery War Axe', 2, CHARACTER_CLASS.WARRIOR, CARD_RARITY.FREE)
def create_weapon(self, v0):
return Weapon(3, 2)
EV P class CLS0(WeaponCard):
def __init__(self):
super().__init__('Fiery War Axe', 2, CHARACTER_CLASS.WARRIOR, CARD_RARITY.FREE,
def create_weapon(self, v0):
return Weapon(3, 2)
EV L class CLS0(MinionCard):
def __init__(self):
super().__init__('Frostwolf Warlord', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Give([Buff(ChangeAttack(Count(MinionSelector()))), Buff(ChangeHealth(Count(MinionSelector())))]), SelfSelector()))
def create_minion(self, v0):
return Minion(4, 4)
EV P class CLS0(MinionCard):
def __init__(self):
super().__init__('Frostwolf Warlord', 5, CHARACTER_CLASS.ALL, CARD_RARITY.COMMON, battlecry=Battlecry(Give([Buff(ChangeAttack(Count(MinionSelector(),), Buff(ChangeHealth(Count(MinionSelector()))))]),), SelfSelector()))
def create_minion(self, v0):
return Minion(4, 4)
EV L class CLS0(SpellCard):
def __init__(self):
super().__init__('Hellfire', 4, CHARACTER_CLASS.WARLOCK, CARD_RARITY.FREE)
def use(self, v0, v1):
super().use(v0, v1)
v2 = copy.copy(v1.other_player.minions)
v2.extend(v1.current_player.minions)
v2.append(v1.other_player.hero)
v2.append(v1.current_player.hero)
for v3 in v2:
v3.damage(v0.effective_spell_damage(3), self)
EV P class CLS0(SpellCard):
def __init__(self):
super().__init__('Hellfire', 4, CHARACTER_CLASS.WARLOCK, CARD_RARITY.FREE,
def use(self, v0, v1):
super().use(v0, v1)
v2 = copy.copy(v1.other_player.minions)
v2.extend(v1.current_player.minions)
for.append(v1.other_player.hero)
for.append(v1.other_player.hero)
for v3 in v2:
.damage(v0.effective_spell_damage(3), self)
```
## Intended uses & limitations
HearthStone card code synthesis.
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 17
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 200
- mixed_precision_training: Native AMP
### Training results
| Training Loss | Epoch | Step | Validation Loss | Exact Match | Bleu | Codebleu | Ngram Match Score | Weighted Ngram Match Score | Syntax Match Score | Dataflow Match Score | Chrf |
|:-------------:|:------:|:-----:|:---------------:|:-----------:|:------:|:--------:|:-----------------:|:--------------------------:|:------------------:|:--------------------:|:-------:|
| 0.8612 | 11.94 | 1600 | 0.2725 | 0.0455 | 0.8477 | 0.6050 | 0.6229 | 0.6335 | 0.6203 | 0.5431 | 88.7010 |
| 0.175 | 23.88 | 3200 | 0.2311 | 0.0909 | 0.8739 | 0.6304 | 0.6566 | 0.6656 | 0.6484 | 0.5508 | 90.7364 |
| 0.1036 | 35.82 | 4800 | 0.2172 | 0.1818 | 0.8930 | 0.6905 | 0.6976 | 0.7062 | 0.7172 | 0.6409 | 91.9702 |
| 0.0695 | 47.76 | 6400 | 0.2233 | 0.2424 | 0.8944 | 0.7017 | 0.7148 | 0.7232 | 0.7187 | 0.6499 | 92.0340 |
| 0.0482 | 59.7 | 8000 | 0.2407 | 0.2879 | 0.9046 | 0.7301 | 0.7387 | 0.7456 | 0.7475 | 0.6885 | 92.6219 |
| 0.0352 | 71.64 | 9600 | 0.2407 | 0.2424 | 0.9074 | 0.7255 | 0.7371 | 0.7448 | 0.7482 | 0.6718 | 92.8281 |
| 0.0262 | 83.58 | 11200 | 0.2596 | 0.3030 | 0.9061 | 0.7445 | 0.7415 | 0.7500 | 0.7774 | 0.7091 | 92.6737 |
| 0.0213 | 95.52 | 12800 | 0.2589 | 0.2879 | 0.9061 | 0.7308 | 0.7409 | 0.7488 | 0.7464 | 0.6873 | 92.7814 |
| 0.0164 | 107.46 | 14400 | 0.2679 | 0.2879 | 0.9096 | 0.7452 | 0.7510 | 0.7592 | 0.7626 | 0.7079 | 92.9900 |
| 0.0131 | 119.4 | 16000 | 0.2660 | 0.2879 | 0.9096 | 0.7447 | 0.7480 | 0.7564 | 0.7666 | 0.7079 | 93.0122 |
| 0.0116 | 131.34 | 17600 | 0.2669 | 0.2727 | 0.9092 | 0.7463 | 0.7445 | 0.7529 | 0.7684 | 0.7194 | 92.9256 |
| 0.0093 | 143.28 | 19200 | 0.2678 | 0.2879 | 0.9113 | 0.7531 | 0.7496 | 0.7581 | 0.7709 | 0.7336 | 93.0406 |
| 0.0083 | 155.22 | 20800 | 0.2728 | 0.2879 | 0.9103 | 0.7407 | 0.7462 | 0.7540 | 0.7702 | 0.6924 | 92.9302 |
| 0.0077 | 167.16 | 22400 | 0.2774 | 0.2879 | 0.9103 | 0.7449 | 0.7449 | 0.7532 | 0.7659 | 0.7156 | 92.9742 |
| 0.0069 | 179.1 | 24000 | 0.2774 | 0.2879 | 0.9120 | 0.7396 | 0.7463 | 0.7539 | 0.7633 | 0.6950 | 93.1057 |
| 0.0069 | 191.04 | 25600 | 0.2782 | 0.2879 | 0.9121 | 0.7482 | 0.7504 | 0.7583 | 0.7673 | 0.7169 | 93.1064 |
### Framework versions
- Transformers 4.24.0
- Pytorch 1.13.0
- Datasets 2.6.1
- Tokenizers 0.13.1
|