upload

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,83 @@
+---
+license: llama2
+base_model: meta-llama/CodeLlama-34b-Instruct-hf
+tags:
+- alignment-handbook
+- generated_from_trainer
+datasets:
+- meng-lab/CodeLlama-34B-Instruct-humaneval
+model-index:
+- name: CodeLlama-34b-Instruct-sft-5e-3-epoch-100-human-eval-final
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/uva-llm/huggingface/runs/93nixyw1)
+# CodeLlama-34b-Instruct-sft-5e-3-epoch-100-human-eval-final
+
+This model is a fine-tuned version of [meta-llama/CodeLlama-34b-Instruct-hf](https://huggingface.co/meta-llama/CodeLlama-34b-Instruct-hf) on the meng-lab/CodeLlama-34B-Instruct-humaneval dataset.
+It achieves the following results on the evaluation set:
+- Loss: 3.7616
+- Loss Layer 6 Head: 1.0709
+- Loss Layer 12 Head: 0.8047
+- Loss Layer 18 Head: 0.7212
+- Loss Layer 24 Head: 0.4396
+- Loss Layer 30 Head: 0.3042
+- Loss Layer 36 Head: 0.2040
+- Loss Layer 42 Head: 0.1346
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.005
+- train_batch_size: 1
+- eval_batch_size: 2
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 128
+- total_eval_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 100
+
+### Training results
+
+| Training Loss | Epoch   | Step | Validation Loss | Loss Layer 6 Head | Loss Layer 12 Head | Loss Layer 18 Head | Loss Layer 24 Head | Loss Layer 30 Head | Loss Layer 36 Head | Loss Layer 42 Head |
+|:-------------:|:-------:|:----:|:---------------:|:-----------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|
+| 3.7028        | 9.1168  | 200  | 4.6352          | 1.4035            | 0.8311             | 1.0429             | 0.4931             | 0.4457             | 0.2349             | 0.1599             |
+| 2.736         | 18.2336 | 400  | 4.7219          | 1.2158            | 0.8316             | 0.7490             | 1.0869             | 0.3238             | 0.2666             | 0.1723             |
+| 2.0128        | 27.3504 | 600  | 3.8953          | 1.1598            | 0.8030             | 0.7230             | 0.4451             | 0.3500             | 0.2027             | 0.1459             |
+| 3.3605        | 36.4672 | 800  | 4.9203          | 1.1038            | 0.8175             | 1.6655             | 0.4410             | 0.3091             | 0.2055             | 0.1365             |
+| 2.5177        | 45.5840 | 1000 | 4.2388          | 1.0907            | 0.8042             | 1.1115             | 0.4403             | 0.3038             | 0.2217             | 0.1412             |
+| 2.0743        | 54.7009 | 1200 | 3.9221          | 1.0727            | 0.8050             | 0.8689             | 0.4418             | 0.3012             | 0.2044             | 0.1362             |
+| 1.8844        | 63.8177 | 1400 | 3.8140          | 1.0723            | 0.8028             | 0.7729             | 0.4389             | 0.3045             | 0.2036             | 0.1350             |
+| 1.8019        | 72.9345 | 1600 | 3.7777          | 1.0726            | 0.8038             | 0.7376             | 0.4401             | 0.3042             | 0.2032             | 0.1345             |
+| 1.7339        | 82.0513 | 1800 | 3.7662          | 1.0703            | 0.8056             | 0.7246             | 0.4394             | 0.3041             | 0.2042             | 0.1347             |
+| 1.6981        | 91.1681 | 2000 | 3.7616          | 1.0709            | 0.8047             | 0.7212             | 0.4396             | 0.3042             | 0.2040             | 0.1346             |
+
+
+### Framework versions
+
+- Transformers 4.43.2
+- Pytorch 2.5.1+cu124
+- Datasets 3.2.0
+- Tokenizers 0.19.1

added_tokens.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fda451030df30d6b04d32f8028647d1f321af096ab028cae5626f98591e3fbf
+size 87

all_results.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18a928486791c757c1e1c3d9197b840ad3f15372ee5416c4f260d6dcc3c89171
+size 248

config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ee43757293d227dced977b2e44aa164695055a87d9cea574b6b1c47ffa11bcc
+size 759

generation_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8efb27b6a9f4c441cb9a76e40fecd2b6273a4bb39b9b23bfde3cb65d64a9297
+size 165

model-00001-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:679949bfffd2f9623f633940999081f545a0edde4e72cd0311921a13dea48b08
+size 4978740880

model-00002-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed3e10b53fc5a5720dcc695e88701bb36d2d71762bc2363e8a18c0f1089325e5
+size 4873882928

model-00003-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1689fe8677db9a3432cd5cf89b2bf89ad2479f915604b552352d29c475c1b80
+size 4815196024

model-00004-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee8b309dbd65e7d6e87e46c1ecb2e39b102254c9304ce7b31f85422e57ada8ba
+size 4873882952

model-00005-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa7f94835f32247b9a492ba1d02d67ca6cca05a5fb2cedf3b9ce1900366505f9
+size 4815196056

model-00006-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f6169284f5fe81ab23accfa0125153c25a94cded2f660030118f58aebc75162
+size 4873882952

model-00007-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ad40e10bfa40896b79a7ff3e3a6380f8a6bde26a7dd020b162fed68e195ced9
+size 4815196056

model-00008-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc715d3442c978e90a70fe4c83909f473c6d2ee150257daf689fcf791a06c3f4
+size 4873882952

model-00009-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:054b1b0621bb7a14b2bd53c1599878340b8642c5e81122bbcf59ecbaed5f1959
+size 4815196056

model-00010-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54414757f8da5530b13b420fac33e8fcff6324aa05d6f21db269da7ec1da95ac
+size 4873882952

model-00011-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88f48912b3a7557533dd8b9a875f1f5768f6085d19e6f261b51e8713cd08b600
+size 4815196056

model-00012-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7992ffad70d218e62473e74917c04e4f7703864edb9ece8bbe0832b98539f41
+size 4873882952

model-00013-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd1d3d7b3a69e0024406617ca43f291af99a517cd3746bd10806dcb2e851657b
+size 4815196056

model-00014-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7acd70c34dc4487a6bbd80f8df76aa8d7fc7dfe94d8f6926757760314b53052
+size 4911648032

model-00015-of-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b19f144cba87b9106f7e068153896fa989e72ce6e7db033d374620d0a02a88f
+size 402653568

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fefe63f5750cd25d75368e6736230d3c2e573c9c9f65ecf8944c72211d6ae9ef
+size 36411

special_tokens_map.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f32ca2dd65336aceb9e359886f34f214330137c9e801e8f41cf542287fc2630
+size 538

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f83209f47d970327175fab9bfb5a05d74da1fff4ada58967223d643e4356e5c
+size 1843531

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ad15334ddd934f433f500267010a3ad3f14497a4a2cde66b6c77a8a5c60ff6f
+size 2652

train_results.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18a928486791c757c1e1c3d9197b840ad3f15372ee5416c4f260d6dcc3c89171
+size 248

trainer_state.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35eb879d6a5ab53b883169db776c4318caf76c75c72e29486e0e8f3a927ee6b0
+size 221811

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:617f52164841430d743a3ae8b7674ff40a8947a4f9915869e0685ac1e0719ee3
+size 6904