Saving train state of step 40

Browse files

Files changed (4) hide show

checkpoint-40-epoch-4/model.safetensors +1 -1
checkpoint-40-epoch-4/optimizer.bin +1 -1
config.json +2 -2
distillation.log +395 -385

checkpoint-40-epoch-4/model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b029dd2ccb30e92e6066f9eddda94088be2d59812663b42f260d6754899a0709
 size 1711916448

 version https://git-lfs.github.com/spec/v1
+oid sha256:344bd2afb9c8d495d0a14fd82190591ea06f1c11c136d2c9be4ffd6dc0edd042
 size 1711916448

checkpoint-40-epoch-4/optimizer.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:62631a72759f0738d2f1477d7011ba2a2f6afb95e84a8366b8b2f28e60c0d65a
 size 962406586

 version https://git-lfs.github.com/spec/v1
+oid sha256:e82a26e4f6b02ddf89b012fe1c02b38e81153d660a31df46c01cb555738ee3d4
 size 962406586

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "biodatlab/distill-whisper-th-medium",
   "activation_dropout": 0.0,
   "activation_function": "gelu",
   "apply_spec_augment": true,
@@ -43,7 +43,7 @@
   "num_mel_bins": 80,
   "pad_token_id": 50257,
   "scale_embedding": false,
-  "torch_dtype": "float32",
   "transformers_version": "4.38.1",
   "use_cache": true,
   "use_weighted_layer_sum": false,

 {
+  "_name_or_path": "distil-whisper/training/distil-whisper-th-medium-4dec",
   "activation_dropout": 0.0,
   "activation_function": "gelu",
   "apply_spec_augment": true,
   "num_mel_bins": 80,
   "pad_token_id": 50257,
   "scale_embedding": false,
+  "torch_dtype": "float16",
   "transformers_version": "4.38.1",
   "use_cache": true,
   "use_weighted_layer_sum": false,

distillation.log CHANGED Viewed

@@ -6,8 +6,8 @@ The following values were not passed to `accelerate launch` and had defaults use
 To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
 /home/hzchng/ai-voicebot/.venv/lib/python3.10/site-packages/accelerate/accelerator.py:401: UserWarning: `log_with=[]` was passed but no supported trackers are currently installed.
   warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
-05/31/2024 01:36:41 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: False
-05/31/2024 01:36:41 - INFO - __main__ - Training/evaluation parameters DistillationTrainingArguments(
 _n_gpu=1,
 accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True},
 adafactor=False,
@@ -80,7 +80,7 @@ local_rank=0,
 log_level=passive,
 log_level_replica=warning,
 log_on_each_node=True,
-logging_dir=/home/hzchng/ai-voicebot/lib/whisper/output/runs/May31_01-36-40_ai-voicebot-training,
 logging_first_step=False,
 logging_nan_inf_filter=True,
 logging_steps=0.1,
@@ -138,16 +138,7 @@ warmup_ratio=0.1,
 warmup_steps=10,
 weight_decay=0.0,
 )
 /home/hzchng/ai-voicebot/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
   warnings.warn(
 loading configuration file config.json from cache at /home/hzchng/.cache/huggingface/hub/models--biodatlab--distill-whisper-th-medium/snapshots/e756107875a3905f3c4535cdca0d499cb56d2232/config.json
@@ -1021,7 +1012,7 @@ Generate config GenerationConfig {
   }
 }
-05/31/2024 01:36:54 - INFO - __main__ - Disabling gradient checkpointing in the decoder since it's incompatible with `freeze_embed_positions`.
 Feature extractor saved in /home/hzchng/ai-voicebot/lib/whisper/output/preprocessor_config.json
 tokenizer config file saved in /home/hzchng/ai-voicebot/lib/whisper/output/tokenizer_config.json
 Special tokens file saved in /home/hzchng/ai-voicebot/lib/whisper/output/special_tokens_map.json
@@ -2684,600 +2675,619 @@ Processor WhisperProcessor:
   "processor_class": "WhisperProcessor"
 }
-05/31/2024 01:36:58 - INFO - __main__ - eval_steps is not set, evaluating at the end of each epoch
-05/31/2024 01:36:59 - INFO - __main__ - ***** Running training *****
-05/31/2024 01:36:59 - INFO - __main__ -   Num examples = 320
-05/31/2024 01:36:59 - INFO - __main__ -   Num epochs = 5
-05/31/2024 01:36:59 - INFO - __main__ -   Instantaneous batch size per device = 8
-05/31/2024 01:36:59 - INFO - __main__ -   Gradient accumulation steps = 1
-05/31/2024 01:36:59 - INFO - __main__ -   Total train batch size (w. parallel & distributed) = 8
-05/31/2024 01:36:59 - INFO - __main__ -   Total optimization steps = 40
-To disable this warning, you can either:
-	- Avoid using `tokenizers` before the fork if possible
-	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
-`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...
-To disable this warning, you can either:
-	- Avoid using `tokenizers` before the fork if possible
-	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
-To disable this warning, you can either:
-	- Avoid using `tokenizers` before the fork if possible
-	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
-To disable this warning, you can either:
-	- Avoid using `tokenizers` before the fork if possible
-	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
-To disable this warning, you can either:
-	- Avoid using `tokenizers` before the fork if possible
-	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
-05/31/2024 01:37:47 - WARNING - accelerate.utils.other - Removed shared tensor {'proj_out.weight'} while saving. This should be OK, but check by verifying that you don't receive any warning while reloading
-05/31/2024 01:37:50 - INFO - accelerate.checkpointing - Model weights saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/model.safetensors
-05/31/2024 01:37:50 - WARNING - accelerate.utils.other - Removed shared tensor {'proj_out.weight'} while saving. This should be OK, but check by verifying that you don't receive any warning while reloading
-05/31/2024 01:37:55 - INFO - accelerate.checkpointing - Model weights saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/model_1.safetensors
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Optimizer state saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/optimizer.bin
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Scheduler state saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/scheduler.bin
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler.bin
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Sampler state for dataloader 1 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_1.bin
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Sampler state for dataloader 2 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_2.bin
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Sampler state for dataloader 3 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_3.bin
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Sampler state for dataloader 4 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_4.bin
-05/31/2024 01:37:58 - INFO - accelerate.checkpointing - Random states saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/random_states_0.pkl
-Some non-default generation parameters are set in the model config. These should go into a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model) instead. This warning will be raised to an exception in v4.41.
-Non-default generation parameters: {'max_length': 448, 'begin_suppress_tokens': [220, 50257]}
-Configuration saved in /home/hzchng/ai-voicebot/lib/whisper/output/config.json
-Configuration saved in /home/hzchng/ai-voicebot/lib/whisper/output/generation_config.json
-Model weights saved in /home/hzchng/ai-voicebot/lib/whisper/output/model.safetensors

 To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
 /home/hzchng/ai-voicebot/.venv/lib/python3.10/site-packages/accelerate/accelerator.py:401: UserWarning: `log_with=[]` was passed but no supported trackers are currently installed.
   warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
+05/31/2024 02:23:34 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: False
+05/31/2024 02:23:34 - INFO - __main__ - Training/evaluation parameters DistillationTrainingArguments(
 _n_gpu=1,
 accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True},
 adafactor=False,
 log_level=passive,
 log_level_replica=warning,
 log_on_each_node=True,
+logging_dir=/home/hzchng/ai-voicebot/lib/whisper/output/runs/May31_02-23-33_ai-voicebot-training,
 logging_first_step=False,
 logging_nan_inf_filter=True,
 logging_steps=0.1,
 warmup_steps=10,
 weight_decay=0.0,
 )
 /home/hzchng/ai-voicebot/.venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
   warnings.warn(
 loading configuration file config.json from cache at /home/hzchng/.cache/huggingface/hub/models--biodatlab--distill-whisper-th-medium/snapshots/e756107875a3905f3c4535cdca0d499cb56d2232/config.json
   }
 }
+05/31/2024 02:23:45 - INFO - __main__ - Disabling gradient checkpointing in the decoder since it's incompatible with `freeze_embed_positions`.
 Feature extractor saved in /home/hzchng/ai-voicebot/lib/whisper/output/preprocessor_config.json
 tokenizer config file saved in /home/hzchng/ai-voicebot/lib/whisper/output/tokenizer_config.json
 Special tokens file saved in /home/hzchng/ai-voicebot/lib/whisper/output/special_tokens_map.json
   "processor_class": "WhisperProcessor"
 }
+05/31/2024 02:23:47 - INFO - __main__ - eval_steps is not set, evaluating at the end of each epoch
+05/31/2024 02:23:48 - INFO - __main__ - ***** Running training *****
+05/31/2024 02:23:48 - INFO - __main__ -   Num examples = 320
+05/31/2024 02:23:48 - INFO - __main__ -   Num epochs = 5
+05/31/2024 02:23:48 - INFO - __main__ -   Instantaneous batch size per device = 8
+05/31/2024 02:23:48 - INFO - __main__ -   Gradient accumulation steps = 1
+05/31/2024 02:23:48 - INFO - __main__ -   Total train batch size (w. parallel & distributed) = 8
+05/31/2024 02:23:48 - INFO - __main__ -   Total optimization steps = 40
+05/31/2024 02:24:35 - WARNING - accelerate.utils.other - Removed shared tensor {'proj_out.weight'} while saving. This should be OK, but check by verifying that you don't receive any warning while reloading
+05/31/2024 02:24:45 - INFO - accelerate.checkpointing - Model weights saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/model.safetensors
+05/31/2024 02:24:45 - WARNING - accelerate.utils.other - Removed shared tensor {'proj_out.weight'} while saving. This should be OK, but check by verifying that you don't receive any warning while reloading
+05/31/2024 02:24:57 - INFO - accelerate.checkpointing - Model weights saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/model_1.safetensors
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Optimizer state saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/optimizer.bin
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Scheduler state saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/scheduler.bin
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler.bin
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Sampler state for dataloader 1 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_1.bin
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Sampler state for dataloader 2 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_2.bin
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Sampler state for dataloader 3 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_3.bin
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Sampler state for dataloader 4 saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/sampler_4.bin
+05/31/2024 02:25:03 - INFO - accelerate.checkpointing - Random states saved in /home/hzchng/ai-voicebot/lib/whisper/output/checkpoint-40-epoch-4/random_states_0.pkl