[2025-01-09 15:54:48,830] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
df: /root/.triton/autotune: No such file or directory
git root error: Cmd('git') failed due to: exit code(128)
  cmdline: git rev-parse --show-toplevel
  stderr: 'fatal: detected dubious ownership in repository at '/workspace'
To add an exception for this directory, call:

	git config --global --add safe.directory /workspace'
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: nguyenducphu201101. Use `wandb login --relogin` to force relogin
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/pydantic/main.py:314: UserWarning: Pydantic serializer warnings:
  Expected `list[str]` but got `tuple` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/pydantic/main.py:314: UserWarning: Pydantic serializer warnings:
  Expected `list[str]` but got `tuple` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
wandb: Tracking run with wandb version 0.19.1
wandb: Run data is saved locally in /workspace/wandb/run-20250109_155451-37f2bdb2-2552-4958-b0be-7186fa7cfbe6
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run test-dpo
wandb: ⭐️ View project at https://wandb.ai/nguyenducphu201101/llm-training-platform
wandb: 🚀 View run at https://wandb.ai/nguyenducphu201101/llm-training-platform/runs/37f2bdb2-2552-4958-b0be-7186fa7cfbe6

Generating train split:   0%|          | 0/1545 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 1545/1545 [00:00<00:00, 119090.67 examples/s]

Generating test split:   0%|          | 0/89 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 89/89 [00:00<00:00, 37679.73 examples/s]
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in '__init__': model_init_kwargs, ref_model_init_kwargs. Will not be supported from version '0.13.0'.

Deprecated positional argument(s) used in DPOTrainer, please use the DPOConfig to set these arguments instead.
  warnings.warn(message, FutureWarning)
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/trl/trainer/dpo_trainer.py:262: UserWarning: You passed `model_init_kwargs` to the DPOTrainer, the value you passed will override the one in the `DPOConfig`.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/trl/trainer/dpo_trainer.py:287: UserWarning: You passed `ref_model_init_kwargs` to the DPOTrainer, the value you passed will override the one in the `DPOConfig`.
  warnings.warn(
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/trl/trainer/dpo_trainer.py:312: UserWarning: You passed a model_id to the DPOTrainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you.
  warnings.warn(
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/trl/trainer/dpo_trainer.py:319: UserWarning: You passed a ref model_id to the DPOTrainer. This will automatically create an `AutoModelForCausalLM`
  warnings.warn(

Extracting prompt from train dataset:   0%|          | 0/1545 [00:00<?, ? examples/s]
Extracting prompt from train dataset:  65%|██████▍   | 1000/1545 [00:00<00:00, 9015.34 examples/s]
Extracting prompt from train dataset: 100%|██████████| 1545/1545 [00:00<00:00, 9401.17 examples/s]

Applying chat template to train dataset:   0%|          | 0/1545 [00:00<?, ? examples/s]
Applying chat template to train dataset:  25%|██▍       | 379/1545 [00:00<00:00, 3744.19 examples/s]
Applying chat template to train dataset:  53%|█████▎    | 812/1545 [00:00<00:00, 4082.95 examples/s]
Applying chat template to train dataset:  92%|█████████▏| 1427/1545 [00:00<00:00, 4085.34 examples/s]
Applying chat template to train dataset: 100%|██████████| 1545/1545 [00:00<00:00, 4014.06 examples/s]

Tokenizing train dataset:   0%|          | 0/1545 [00:00<?, ? examples/s]
Tokenizing train dataset:   7%|▋         | 110/1545 [00:00<00:01, 1087.62 examples/s]
Tokenizing train dataset:  15%|█▌        | 232/1545 [00:00<00:01, 1153.16 examples/s]
Tokenizing train dataset:  23%|██▎       | 353/1545 [00:00<00:01, 1170.38 examples/s]
Tokenizing train dataset:  31%|███       | 472/1545 [00:00<00:00, 1172.74 examples/s]
Tokenizing train dataset:  41%|████▏     | 639/1545 [00:00<00:00, 1140.62 examples/s]
Tokenizing train dataset:  52%|█████▏    | 810/1545 [00:00<00:00, 1133.80 examples/s]
Tokenizing train dataset:  60%|██████    | 929/1545 [00:00<00:00, 1143.19 examples/s]
Tokenizing train dataset:  68%|██████▊   | 1051/1545 [00:00<00:00, 1161.64 examples/s]
Tokenizing train dataset:  76%|███████▌  | 1176/1545 [00:01<00:00, 1183.67 examples/s]
Tokenizing train dataset:  88%|████████▊ | 1354/1545 [00:01<00:00, 1182.12 examples/s]
Tokenizing train dataset:  99%|█████████▉| 1530/1545 [00:01<00:00, 1173.45 examples/s]
Tokenizing train dataset: 100%|██████████| 1545/1545 [00:01<00:00, 1158.17 examples/s]
wandb: WARNING The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.

  0%|          | 0/1545 [00:00<?, ?it/s]Could not estimate the number of tokens of the input, floating-point operations will not be computed

  0%|          | 1/1545 [00:01<30:08,  1.17s/it]
  0%|          | 2/1545 [00:02<25:15,  1.02it/s]
  0%|          | 3/1545 [00:02<17:59,  1.43it/s]
  0%|          | 4/1545 [00:02<16:30,  1.56it/s]
  0%|          | 5/1545 [00:03<14:43,  1.74it/s]
  0%|          | 6/1545 [00:03<14:45,  1.74it/s]
  0%|          | 7/1545 [00:04<14:57,  1.71it/s]
  1%|          | 8/1545 [00:05<15:12,  1.68it/s]
  1%|          | 9/1545 [00:05<15:30,  1.65it/s]
  1%|          | 10/1545 [00:06<15:11,  1.68it/s]
                                                 
{'loss': 2.7692, 'grad_norm': 18.5, 'learning_rate': 9.935275080906149e-06, 'rewards/chosen': -13.267723083496094, 'rewards/rejected': -12.376993179321289, 'rewards/accuracies': 0.5, 'rewards/margins': -0.8907286524772644, 'logps/chosen': -272.36419677734375, 'logps/rejected': -224.4181365966797, 'logits/chosen': -0.7734757661819458, 'logits/rejected': -0.8499571084976196, 'epoch': 0.01}

  1%|          | 10/1545 [00:06<15:11,  1.68it/s]
  1%|          | 11/1545 [00:06<14:49,  1.72it/s]
  1%|          | 12/1545 [00:07<14:17,  1.79it/s]
  1%|          | 13/1545 [00:08<14:34,  1.75it/s]
  1%|          | 14/1545 [00:08<14:24,  1.77it/s]
  1%|          | 15/1545 [00:09<13:25,  1.90it/s]
  1%|          | 16/1545 [00:09<13:46,  1.85it/s]
  1%|          | 17/1545 [00:10<13:56,  1.83it/s]
  1%|          | 18/1545 [00:10<14:00,  1.82it/s]
  1%|          | 19/1545 [00:11<13:08,  1.94it/s]
  1%|▏         | 20/1545 [00:11<13:36,  1.87it/s]
                                                 
{'loss': 0.2762, 'grad_norm': 0.8828125, 'learning_rate': 9.870550161812299e-06, 'rewards/chosen': -16.860511779785156, 'rewards/rejected': -22.893171310424805, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 6.03265905380249, 'logps/chosen': -311.78521728515625, 'logps/rejected': -344.44610595703125, 'logits/chosen': -0.5504701137542725, 'logits/rejected': -0.5987938642501831, 'epoch': 0.01}

  1%|▏         | 20/1545 [00:11<13:36,  1.87it/s]
  1%|▏         | 21/1545 [00:12<14:05,  1.80it/s]
  1%|▏         | 22/1545 [00:12<13:28,  1.88it/s]
  1%|▏         | 23/1545 [00:13<13:57,  1.82it/s]
  2%|▏         | 24/1545 [00:14<14:16,  1.78it/s]
  2%|▏         | 25/1545 [00:14<14:12,  1.78it/s]
  2%|▏         | 26/1545 [00:15<13:29,  1.88it/s]
  2%|▏         | 27/1545 [00:15<13:49,  1.83it/s]
  2%|▏         | 28/1545 [00:16<13:54,  1.82it/s]
  2%|▏         | 29/1545 [00:16<13:13,  1.91it/s]
  2%|▏         | 30/1545 [00:17<13:19,  1.89it/s]
                                                 
{'loss': 4.4466, 'grad_norm': 7.82310962677002e-08, 'learning_rate': 9.805825242718447e-06, 'rewards/chosen': -35.80329132080078, 'rewards/rejected': -39.130760192871094, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 3.3274664878845215, 'logps/chosen': -513.8514404296875, 'logps/rejected': -501.32061767578125, 'logits/chosen': -2.7659506797790527, 'logits/rejected': -3.2069015502929688, 'epoch': 0.02}

  2%|▏         | 30/1545 [00:17<13:19,  1.89it/s]
  2%|▏         | 31/1545 [00:17<13:40,  1.84it/s]
  2%|▏         | 32/1545 [00:18<13:52,  1.82it/s]
  2%|▏         | 33/1545 [00:18<12:56,  1.95it/s]
  2%|▏         | 34/1545 [00:19<11:55,  2.11it/s]
  2%|▏         | 35/1545 [00:19<12:51,  1.96it/s]
  2%|▏         | 36/1545 [00:20<13:24,  1.88it/s]
  2%|▏         | 37/1545 [00:20<12:32,  2.00it/s]
  2%|▏         | 38/1545 [00:21<13:11,  1.90it/s]
  3%|▎         | 39/1545 [00:21<13:24,  1.87it/s]
  3%|▎         | 40/1545 [00:22<13:16,  1.89it/s]
                                                 
{'loss': 1.966, 'grad_norm': 1616.0, 'learning_rate': 9.741100323624596e-06, 'rewards/chosen': -40.694602966308594, 'rewards/rejected': -51.3997802734375, 'rewards/accuracies': 0.6000000238418579, 'rewards/margins': 10.705179214477539, 'logps/chosen': -561.7604370117188, 'logps/rejected': -636.379638671875, 'logits/chosen': -2.4068899154663086, 'logits/rejected': -2.463327407836914, 'epoch': 0.03}

  3%|▎         | 40/1545 [00:22<13:16,  1.89it/s]
  3%|▎         | 41/1545 [00:22<13:18,  1.88it/s]
  3%|▎         | 42/1545 [00:23<13:33,  1.85it/s]
  3%|▎         | 43/1545 [00:23<12:27,  2.01it/s]
  3%|▎         | 44/1545 [00:24<12:09,  2.06it/s]
  3%|▎         | 45/1545 [00:24<12:41,  1.97it/s]
  3%|▎         | 46/1545 [00:25<13:09,  1.90it/s]
  3%|▎         | 47/1545 [00:26<13:23,  1.86it/s]
  3%|▎         | 48/1545 [00:26<12:13,  2.04it/s]
  3%|▎         | 49/1545 [00:27<12:59,  1.92it/s]
  3%|▎         | 50/1545 [00:27<13:16,  1.88it/s]
                                                 
{'loss': 2.376, 'grad_norm': 26.75, 'learning_rate': 9.676375404530746e-06, 'rewards/chosen': -23.04972267150879, 'rewards/rejected': -31.14634132385254, 'rewards/accuracies': 0.6000000238418579, 'rewards/margins': 8.09661865234375, 'logps/chosen': -366.2521057128906, 'logps/rejected': -414.253173828125, 'logits/chosen': -1.4296232461929321, 'logits/rejected': -1.88992440700531, 'epoch': 0.03}

  3%|▎         | 50/1545 [00:27<13:16,  1.88it/s]
  3%|▎         | 51/1545 [00:28<13:31,  1.84it/s]
  3%|▎         | 52/1545 [00:28<13:15,  1.88it/s]
  3%|▎         | 53/1545 [00:29<13:37,  1.82it/s]
  3%|▎         | 54/1545 [00:29<13:40,  1.82it/s]
  4%|▎         | 55/1545 [00:30<12:53,  1.93it/s]
  4%|▎         | 56/1545 [00:30<13:25,  1.85it/s]
  4%|▎         | 57/1545 [00:31<13:30,  1.84it/s]
  4%|▍         | 58/1545 [00:31<13:25,  1.85it/s]
  4%|▍         | 59/1545 [00:32<13:04,  1.89it/s]
  4%|▍         | 60/1545 [00:32<13:26,  1.84it/s]
                                                 
{'loss': 3.5255, 'grad_norm': 1.2620790914089075e-19, 'learning_rate': 9.611650485436894e-06, 'rewards/chosen': -41.635231018066406, 'rewards/rejected': -57.55849075317383, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 15.923251152038574, 'logps/chosen': -561.2437744140625, 'logps/rejected': -689.3187255859375, 'logits/chosen': -3.666018009185791, 'logits/rejected': -3.4739983081817627, 'epoch': 0.04}

  4%|▍         | 60/1545 [00:33<13:26,  1.84it/s]
  4%|▍         | 61/1545 [00:33<13:25,  1.84it/s]
  4%|▍         | 62/1545 [00:33<12:48,  1.93it/s]
  4%|▍         | 63/1545 [00:34<13:15,  1.86it/s]
  4%|▍         | 64/1545 [00:36<21:48,  1.13it/s]
  4%|▍         | 65/1545 [00:36<19:19,  1.28it/s]
  4%|▍         | 66/1545 [00:37<17:18,  1.42it/s]
  4%|▍         | 67/1545 [00:37<16:28,  1.50it/s]
  4%|▍         | 68/1545 [00:38<15:40,  1.57it/s]
  4%|▍         | 69/1545 [00:38<14:16,  1.72it/s]
  5%|▍         | 70/1545 [00:39<14:19,  1.72it/s]
                                                 
{'loss': 0.4333, 'grad_norm': 4000.0, 'learning_rate': 9.546925566343042e-06, 'rewards/chosen': -40.17411422729492, 'rewards/rejected': -62.64879608154297, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 22.47468376159668, 'logps/chosen': -535.5774536132812, 'logps/rejected': -740.8128662109375, 'logits/chosen': -3.367959976196289, 'logits/rejected': -3.1139683723449707, 'epoch': 0.05}

  5%|▍         | 70/1545 [00:39<14:19,  1.72it/s]
  5%|▍         | 71/1545 [00:40<14:10,  1.73it/s]
  5%|▍         | 72/1545 [00:40<13:52,  1.77it/s]
  5%|▍         | 73/1545 [00:40<12:05,  2.03it/s]
  5%|▍         | 74/1545 [00:41<12:40,  1.93it/s]
  5%|▍         | 75/1545 [00:42<13:09,  1.86it/s]
  5%|▍         | 76/1545 [00:42<13:07,  1.87it/s]
  5%|▍         | 77/1545 [00:43<13:09,  1.86it/s]
  5%|▌         | 78/1545 [00:43<13:23,  1.83it/s]
  5%|▌         | 79/1545 [00:44<13:44,  1.78it/s]
  5%|▌         | 80/1545 [00:44<12:42,  1.92it/s]
                                                 
{'loss': 0.4326, 'grad_norm': 13824.0, 'learning_rate': 9.482200647249192e-06, 'rewards/chosen': -48.43840789794922, 'rewards/rejected': -76.73748779296875, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 28.299081802368164, 'logps/chosen': -626.9752807617188, 'logps/rejected': -873.7962646484375, 'logits/chosen': -2.483962297439575, 'logits/rejected': -2.444535493850708, 'epoch': 0.05}

  5%|▌         | 80/1545 [00:44<12:42,  1.92it/s]
  5%|▌         | 81/1545 [00:45<13:20,  1.83it/s]
  5%|▌         | 82/1545 [00:45<13:23,  1.82it/s]
  5%|▌         | 83/1545 [00:46<13:11,  1.85it/s]
  5%|▌         | 84/1545 [00:46<12:54,  1.89it/s]
  6%|▌         | 85/1545 [00:47<13:07,  1.85it/s]
  6%|▌         | 86/1545 [00:48<13:24,  1.81it/s]
  6%|▌         | 87/1545 [00:48<12:57,  1.87it/s]
  6%|▌         | 88/1545 [00:49<13:13,  1.84it/s]
  6%|▌         | 89/1545 [00:49<13:20,  1.82it/s]
  6%|▌         | 90/1545 [00:50<13:17,  1.82it/s]
                                                 
{'loss': 2.8932, 'grad_norm': 0.001190185546875, 'learning_rate': 9.41747572815534e-06, 'rewards/chosen': -35.804256439208984, 'rewards/rejected': -56.7745361328125, 'rewards/accuracies': 0.699999988079071, 'rewards/margins': 20.97027015686035, 'logps/chosen': -504.11846923828125, 'logps/rejected': -688.4282836914062, 'logits/chosen': -1.8350107669830322, 'logits/rejected': -1.7690149545669556, 'epoch': 0.06}

  6%|▌         | 90/1545 [00:50<13:17,  1.82it/s]
  6%|▌         | 91/1545 [00:50<13:03,  1.86it/s]
  6%|▌         | 92/1545 [00:51<13:32,  1.79it/s]
  6%|▌         | 93/1545 [00:51<13:47,  1.76it/s]
  6%|▌         | 94/1545 [00:52<12:27,  1.94it/s]
  6%|▌         | 95/1545 [00:52<11:45,  2.06it/s]
  6%|▌         | 96/1545 [00:53<12:24,  1.95it/s]
  6%|▋         | 97/1545 [00:53<12:46,  1.89it/s]
  6%|▋         | 98/1545 [00:54<12:21,  1.95it/s]
  6%|▋         | 99/1545 [00:54<12:47,  1.88it/s]
  6%|▋         | 100/1545 [00:55<13:04,  1.84it/s]
                                                  
{'loss': 1.493, 'grad_norm': 0.055908203125, 'learning_rate': 9.35275080906149e-06, 'rewards/chosen': -16.02591896057129, 'rewards/rejected': -22.183359146118164, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 6.157437801361084, 'logps/chosen': -340.04986572265625, 'logps/rejected': -332.9122314453125, 'logits/chosen': -1.376056432723999, 'logits/rejected': -1.7216821908950806, 'epoch': 0.06}

  6%|▋         | 100/1545 [00:55<13:04,  1.84it/s]
  7%|▋         | 101/1545 [00:56<13:28,  1.79it/s]
  7%|▋         | 102/1545 [00:56<12:21,  1.95it/s]
  7%|▋         | 103/1545 [00:57<12:58,  1.85it/s]
  7%|▋         | 104/1545 [00:57<13:06,  1.83it/s]
  7%|▋         | 105/1545 [00:58<11:49,  2.03it/s]
  7%|▋         | 106/1545 [00:58<11:20,  2.11it/s]
  7%|▋         | 107/1545 [00:59<12:10,  1.97it/s]
  7%|▋         | 108/1545 [00:59<11:12,  2.14it/s]
  7%|▋         | 109/1545 [01:00<11:50,  2.02it/s]
  7%|▋         | 110/1545 [01:00<11:33,  2.07it/s]
                                                  
{'loss': 1.7504, 'grad_norm': 68.5, 'learning_rate': 9.288025889967638e-06, 'rewards/chosen': -17.503616333007812, 'rewards/rejected': -20.149761199951172, 'rewards/accuracies': 0.5, 'rewards/margins': 2.6461453437805176, 'logps/chosen': -324.0485534667969, 'logps/rejected': -315.428955078125, 'logits/chosen': -1.0650968551635742, 'logits/rejected': -1.4754924774169922, 'epoch': 0.07}

  7%|▋         | 110/1545 [01:00<11:33,  2.07it/s]
  7%|▋         | 111/1545 [01:01<12:15,  1.95it/s]
  7%|▋         | 112/1545 [01:01<12:51,  1.86it/s]
  7%|▋         | 113/1545 [01:02<12:17,  1.94it/s]
  7%|▋         | 114/1545 [01:02<11:17,  2.11it/s]
  7%|▋         | 115/1545 [01:03<12:05,  1.97it/s]
  8%|▊         | 116/1545 [01:03<12:24,  1.92it/s]
  8%|▊         | 117/1545 [01:03<11:06,  2.14it/s]
  8%|▊         | 118/1545 [01:04<11:36,  2.05it/s]
  8%|▊         | 119/1545 [01:05<12:12,  1.95it/s]
  8%|▊         | 120/1545 [01:05<11:08,  2.13it/s]
                                                  
{'loss': 0.0812, 'grad_norm': 322.0, 'learning_rate': 9.223300970873788e-06, 'rewards/chosen': -22.525089263916016, 'rewards/rejected': -34.67645263671875, 'rewards/accuracies': 1.0, 'rewards/margins': 12.151366233825684, 'logps/chosen': -371.64007568359375, 'logps/rejected': -470.7439880371094, 'logits/chosen': -3.351106643676758, 'logits/rejected': -3.953688144683838, 'epoch': 0.08}

  8%|▊         | 120/1545 [01:05<11:08,  2.13it/s]
  8%|▊         | 121/1545 [01:05<10:27,  2.27it/s]
  8%|▊         | 122/1545 [01:06<10:18,  2.30it/s]
  8%|▊         | 123/1545 [01:06<11:14,  2.11it/s]
  8%|▊         | 124/1545 [01:07<11:40,  2.03it/s]
  8%|▊         | 125/1545 [01:07<12:05,  1.96it/s]
  8%|▊         | 126/1545 [01:08<11:53,  1.99it/s]
  8%|▊         | 127/1545 [01:08<12:14,  1.93it/s]
  8%|▊         | 128/1545 [01:09<11:14,  2.10it/s]
  8%|▊         | 129/1545 [01:09<11:30,  2.05it/s]
  8%|▊         | 130/1545 [01:10<11:49,  1.99it/s]
                                                  
{'loss': 8.072, 'grad_norm': 7.534027099609375e-05, 'learning_rate': 9.158576051779936e-06, 'rewards/chosen': -28.951553344726562, 'rewards/rejected': -31.89861488342285, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 2.9470643997192383, 'logps/chosen': -441.2911682128906, 'logps/rejected': -428.54656982421875, 'logits/chosen': -3.3049476146698, 'logits/rejected': -3.9077095985412598, 'epoch': 0.08}

  8%|▊         | 130/1545 [01:10<11:49,  1.99it/s]
  8%|▊         | 131/1545 [01:10<12:16,  1.92it/s]
  9%|▊         | 132/1545 [01:11<12:20,  1.91it/s]
  9%|▊         | 133/1545 [01:11<11:28,  2.05it/s]
  9%|▊         | 134/1545 [01:12<12:02,  1.95it/s]
  9%|▊         | 135/1545 [01:13<12:24,  1.89it/s]
  9%|▉         | 136/1545 [01:13<12:28,  1.88it/s]
  9%|▉         | 137/1545 [01:13<11:41,  2.01it/s]
  9%|▉         | 138/1545 [01:14<12:11,  1.92it/s]
  9%|▉         | 139/1545 [01:15<12:24,  1.89it/s]
  9%|▉         | 140/1545 [01:15<12:20,  1.90it/s]
                                                  
{'loss': 0.9608, 'grad_norm': 2.086162567138672e-07, 'learning_rate': 9.093851132686085e-06, 'rewards/chosen': -19.53597640991211, 'rewards/rejected': -32.082984924316406, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 12.547004699707031, 'logps/chosen': -317.4342956542969, 'logps/rejected': -419.8863220214844, 'logits/chosen': -2.860546112060547, 'logits/rejected': -3.551792621612549, 'epoch': 0.09}

  9%|▉         | 140/1545 [01:15<12:20,  1.90it/s]
  9%|▉         | 141/1545 [01:15<11:08,  2.10it/s]
  9%|▉         | 142/1545 [01:16<11:58,  1.95it/s]
  9%|▉         | 143/1545 [01:17<12:16,  1.90it/s]
  9%|▉         | 144/1545 [01:17<12:07,  1.93it/s]
  9%|▉         | 145/1545 [01:18<12:27,  1.87it/s]
  9%|▉         | 146/1545 [01:18<12:40,  1.84it/s]
 10%|▉         | 147/1545 [01:19<12:40,  1.84it/s]
 10%|▉         | 148/1545 [01:19<11:53,  1.96it/s]
 10%|▉         | 149/1545 [01:20<12:18,  1.89it/s]
 10%|▉         | 150/1545 [01:20<12:33,  1.85it/s]
                                                  
{'loss': 1.0182, 'grad_norm': 18.5, 'learning_rate': 9.029126213592233e-06, 'rewards/chosen': -17.807960510253906, 'rewards/rejected': -36.81556701660156, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 19.00760269165039, 'logps/chosen': -303.423095703125, 'logps/rejected': -467.78778076171875, 'logits/chosen': -3.5883376598358154, 'logits/rejected': -4.601079940795898, 'epoch': 0.1}

 10%|▉         | 150/1545 [01:20<12:33,  1.85it/s]
 10%|▉         | 151/1545 [01:21<12:15,  1.89it/s]
 10%|▉         | 152/1545 [01:21<12:18,  1.89it/s]
 10%|▉         | 153/1545 [01:22<12:30,  1.85it/s]
 10%|▉         | 154/1545 [01:23<12:41,  1.83it/s]
 10%|█         | 155/1545 [01:23<12:00,  1.93it/s]
 10%|█         | 156/1545 [01:24<12:31,  1.85it/s]
 10%|█         | 157/1545 [01:24<11:22,  2.03it/s]
 10%|█         | 158/1545 [01:25<11:47,  1.96it/s]
 10%|█         | 159/1545 [01:25<11:12,  2.06it/s]
 10%|█         | 160/1545 [01:26<11:57,  1.93it/s]
                                                  
{'loss': 0.0122, 'grad_norm': 6.184563972055912e-10, 'learning_rate': 8.964401294498383e-06, 'rewards/chosen': -19.044435501098633, 'rewards/rejected': -41.322547912597656, 'rewards/accuracies': 1.0, 'rewards/margins': 22.278114318847656, 'logps/chosen': -330.6968994140625, 'logps/rejected': -523.6119384765625, 'logits/chosen': -3.189589023590088, 'logits/rejected': -4.832894325256348, 'epoch': 0.1}

 10%|█         | 160/1545 [01:26<11:57,  1.93it/s]
 10%|█         | 161/1545 [01:26<12:13,  1.89it/s]
 10%|█         | 162/1545 [01:27<12:12,  1.89it/s]
 11%|█         | 163/1545 [01:27<12:04,  1.91it/s]
 11%|█         | 164/1545 [01:28<12:25,  1.85it/s]
 11%|█         | 165/1545 [01:28<12:29,  1.84it/s]
 11%|█         | 166/1545 [01:29<11:56,  1.92it/s]
 11%|█         | 167/1545 [01:29<12:14,  1.88it/s]
 11%|█         | 168/1545 [01:30<12:20,  1.86it/s]
 11%|█         | 169/1545 [01:30<12:19,  1.86it/s]
 11%|█         | 170/1545 [01:31<12:00,  1.91it/s]
                                                  
{'loss': 1.4695, 'grad_norm': 8.216219588366713e-20, 'learning_rate': 8.899676375404531e-06, 'rewards/chosen': -46.257057189941406, 'rewards/rejected': -82.32180786132812, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 36.06475067138672, 'logps/chosen': -610.6324462890625, 'logps/rejected': -940.2429809570312, 'logits/chosen': -4.580934047698975, 'logits/rejected': -5.433409214019775, 'epoch': 0.11}

 11%|█         | 170/1545 [01:31<12:00,  1.91it/s]
 11%|█         | 171/1545 [01:31<12:26,  1.84it/s]
 11%|█         | 172/1545 [01:32<12:13,  1.87it/s]
 11%|█         | 173/1545 [01:32<11:43,  1.95it/s]
 11%|█▏        | 174/1545 [01:33<12:17,  1.86it/s]
 11%|█▏        | 175/1545 [01:34<12:30,  1.82it/s]
 11%|█▏        | 176/1545 [01:34<12:26,  1.83it/s]
 11%|█▏        | 177/1545 [01:34<10:43,  2.13it/s]
 12%|█▏        | 178/1545 [01:35<11:25,  1.99it/s]
 12%|█▏        | 179/1545 [01:36<11:43,  1.94it/s]
 12%|█▏        | 180/1545 [01:37<16:58,  1.34it/s]
                                                  
{'loss': 1.1198, 'grad_norm': 0.03271484375, 'learning_rate': 8.834951456310681e-06, 'rewards/chosen': -24.422029495239258, 'rewards/rejected': -46.21000289916992, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 21.787975311279297, 'logps/chosen': -383.09124755859375, 'logps/rejected': -577.4222412109375, 'logits/chosen': -4.082339286804199, 'logits/rejected': -5.3761396408081055, 'epoch': 0.12}

 12%|█▏        | 180/1545 [01:37<16:58,  1.34it/s]
 12%|█▏        | 181/1545 [01:37<15:05,  1.51it/s]
 12%|█▏        | 182/1545 [01:38<14:34,  1.56it/s]
 12%|█▏        | 183/1545 [01:38<12:40,  1.79it/s]
 12%|█▏        | 184/1545 [01:39<12:34,  1.80it/s]
 12%|█▏        | 185/1545 [01:39<12:23,  1.83it/s]
 12%|█▏        | 186/1545 [01:40<12:33,  1.80it/s]
 12%|█▏        | 187/1545 [01:40<12:35,  1.80it/s]
 12%|█▏        | 188/1545 [01:41<11:44,  1.93it/s]
 12%|█▏        | 189/1545 [01:41<12:01,  1.88it/s]
 12%|█▏        | 190/1545 [01:42<12:12,  1.85it/s]
                                                  
{'loss': 0.0704, 'grad_norm': 0.00084686279296875, 'learning_rate': 8.770226537216829e-06, 'rewards/chosen': -19.553829193115234, 'rewards/rejected': -37.149200439453125, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 17.59537124633789, 'logps/chosen': -342.45489501953125, 'logps/rejected': -478.92120361328125, 'logits/chosen': -3.0818886756896973, 'logits/rejected': -4.275086402893066, 'epoch': 0.12}

 12%|█▏        | 190/1545 [01:42<12:12,  1.85it/s]
 12%|█▏        | 191/1545 [01:43<12:16,  1.84it/s]
 12%|█▏        | 192/1545 [01:43<11:25,  1.97it/s]
 12%|█▏        | 193/1545 [01:44<11:49,  1.90it/s]
 13%|█▎        | 194/1545 [01:44<12:07,  1.86it/s]
 13%|█▎        | 195/1545 [01:44<10:58,  2.05it/s]
 13%|█▎        | 196/1545 [01:45<10:29,  2.14it/s]
 13%|█▎        | 197/1545 [01:45<11:07,  2.02it/s]
 13%|█▎        | 198/1545 [01:46<11:23,  1.97it/s]
 13%|█▎        | 199/1545 [01:47<11:34,  1.94it/s]
 13%|█▎        | 200/1545 [01:47<11:37,  1.93it/s]
                                                  
{'loss': 0.012, 'grad_norm': 6.0625, 'learning_rate': 8.705501618122979e-06, 'rewards/chosen': -33.0775146484375, 'rewards/rejected': -53.18238067626953, 'rewards/accuracies': 1.0, 'rewards/margins': 20.104867935180664, 'logps/chosen': -470.367919921875, 'logps/rejected': -644.5907592773438, 'logits/chosen': -3.6899101734161377, 'logits/rejected': -5.173361778259277, 'epoch': 0.13}

 13%|█▎        | 200/1545 [01:47<11:37,  1.93it/s]
 13%|█▎        | 201/1545 [01:48<12:10,  1.84it/s]
 13%|█▎        | 202/1545 [01:48<12:21,  1.81it/s]
 13%|█▎        | 203/1545 [01:49<11:27,  1.95it/s]
 13%|█▎        | 204/1545 [01:49<11:50,  1.89it/s]
 13%|█▎        | 205/1545 [01:50<12:17,  1.82it/s]
 13%|█▎        | 206/1545 [01:50<12:13,  1.83it/s]
 13%|█▎        | 207/1545 [01:51<12:05,  1.84it/s]
 13%|█▎        | 208/1545 [01:51<12:23,  1.80it/s]
 14%|█▎        | 209/1545 [01:52<12:19,  1.81it/s]
 14%|█▎        | 210/1545 [01:53<11:46,  1.89it/s]
                                                  
{'loss': 0.1499, 'grad_norm': 0.0, 'learning_rate': 8.640776699029127e-06, 'rewards/chosen': -35.89064025878906, 'rewards/rejected': -85.90962219238281, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 50.01898956298828, 'logps/chosen': -512.2362670898438, 'logps/rejected': -987.8312377929688, 'logits/chosen': -3.8464527130126953, 'logits/rejected': -5.528945446014404, 'epoch': 0.14}

 14%|█▎        | 210/1545 [01:53<11:46,  1.89it/s]
 14%|█▎        | 211/1545 [01:53<12:16,  1.81it/s]
 14%|█▎        | 212/1545 [01:54<12:23,  1.79it/s]
 14%|█▍        | 213/1545 [01:54<12:12,  1.82it/s]
 14%|█▍        | 214/1545 [01:55<11:54,  1.86it/s]
 14%|█▍        | 215/1545 [01:55<12:10,  1.82it/s]
 14%|█▍        | 216/1545 [01:56<12:09,  1.82it/s]
 14%|█▍        | 217/1545 [01:56<11:13,  1.97it/s]
 14%|█▍        | 218/1545 [01:57<11:42,  1.89it/s]
 14%|█▍        | 219/1545 [01:57<11:54,  1.86it/s]
 14%|█▍        | 220/1545 [01:58<11:52,  1.86it/s]
                                                  
{'loss': 0.6222, 'grad_norm': 720.0, 'learning_rate': 8.576051779935276e-06, 'rewards/chosen': -39.93156814575195, 'rewards/rejected': -58.58339309692383, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 18.65182876586914, 'logps/chosen': -554.5364990234375, 'logps/rejected': -703.7771606445312, 'logits/chosen': -3.3229453563690186, 'logits/rejected': -4.664990425109863, 'epoch': 0.14}

 14%|█▍        | 220/1545 [01:58<11:52,  1.86it/s]
 14%|█▍        | 221/1545 [01:58<11:11,  1.97it/s]
 14%|█▍        | 222/1545 [01:59<11:35,  1.90it/s]
 14%|█▍        | 223/1545 [02:00<11:46,  1.87it/s]
 14%|█▍        | 224/1545 [02:00<10:40,  2.06it/s]
 15%|█▍        | 225/1545 [02:00<10:18,  2.13it/s]
 15%|█▍        | 226/1545 [02:01<11:03,  1.99it/s]
 15%|█▍        | 227/1545 [02:01<10:09,  2.16it/s]
 15%|█▍        | 228/1545 [02:02<10:43,  2.05it/s]
 15%|█▍        | 229/1545 [02:02<09:23,  2.33it/s]
 15%|█▍        | 230/1545 [02:03<10:25,  2.10it/s]
                                                  
{'loss': 0.0693, 'grad_norm': 0.0, 'learning_rate': 8.511326860841424e-06, 'rewards/chosen': -45.17763900756836, 'rewards/rejected': -84.54122924804688, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 39.363590240478516, 'logps/chosen': -602.5833740234375, 'logps/rejected': -968.5574951171875, 'logits/chosen': -3.7494091987609863, 'logits/rejected': -5.287755966186523, 'epoch': 0.15}

 15%|█▍        | 230/1545 [02:03<10:25,  2.10it/s]
 15%|█▍        | 231/1545 [02:03<11:00,  1.99it/s]
 15%|█▌        | 232/1545 [02:04<11:10,  1.96it/s]
 15%|█▌        | 233/1545 [02:04<11:03,  1.98it/s]
 15%|█▌        | 234/1545 [02:05<11:36,  1.88it/s]
 15%|█▌        | 235/1545 [02:05<11:29,  1.90it/s]
 15%|█▌        | 236/1545 [02:06<11:04,  1.97it/s]
 15%|█▌        | 237/1545 [02:06<11:44,  1.86it/s]
 15%|█▌        | 238/1545 [02:07<11:56,  1.82it/s]
 15%|█▌        | 239/1545 [02:08<11:52,  1.83it/s]
 16%|█▌        | 240/1545 [02:08<10:52,  2.00it/s]
                                                  
{'loss': 0.542, 'grad_norm': 334.0, 'learning_rate': 8.446601941747573e-06, 'rewards/chosen': -34.770347595214844, 'rewards/rejected': -57.98395919799805, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 23.213611602783203, 'logps/chosen': -471.40185546875, 'logps/rejected': -686.8424682617188, 'logits/chosen': -3.1056551933288574, 'logits/rejected': -4.154791355133057, 'epoch': 0.16}

 16%|█▌        | 240/1545 [02:08<10:52,  2.00it/s]
 16%|█▌        | 241/1545 [02:09<11:23,  1.91it/s]
 16%|█▌        | 242/1545 [02:09<11:28,  1.89it/s]
 16%|█▌        | 243/1545 [02:10<11:26,  1.90it/s]
 16%|█▌        | 244/1545 [02:10<11:16,  1.92it/s]
 16%|█▌        | 245/1545 [02:11<11:35,  1.87it/s]
 16%|█▌        | 246/1545 [02:11<11:44,  1.84it/s]
 16%|█▌        | 247/1545 [02:12<11:09,  1.94it/s]
 16%|█▌        | 248/1545 [02:12<11:29,  1.88it/s]
 16%|█▌        | 249/1545 [02:13<11:42,  1.85it/s]
 16%|█▌        | 250/1545 [02:13<11:35,  1.86it/s]
                                                  
{'loss': 0.0827, 'grad_norm': 0.00689697265625, 'learning_rate': 8.381877022653722e-06, 'rewards/chosen': -13.427328109741211, 'rewards/rejected': -41.39923095703125, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 27.97190284729004, 'logps/chosen': -279.3583068847656, 'logps/rejected': -531.146484375, 'logits/chosen': -1.5291231870651245, 'logits/rejected': -3.7414169311523438, 'epoch': 0.16}

 16%|█▌        | 250/1545 [02:13<11:35,  1.86it/s]
 16%|█▌        | 251/1545 [02:14<11:04,  1.95it/s]
 16%|█▋        | 252/1545 [02:14<11:31,  1.87it/s]
 16%|█▋        | 253/1545 [02:15<10:22,  2.08it/s]
 16%|█▋        | 254/1545 [02:15<10:44,  2.00it/s]
 17%|█▋        | 255/1545 [02:16<10:06,  2.13it/s]
 17%|█▋        | 256/1545 [02:16<10:46,  2.00it/s]
 17%|█▋        | 257/1545 [02:17<11:12,  1.92it/s]
 17%|█▋        | 258/1545 [02:17<11:10,  1.92it/s]
 17%|█▋        | 259/1545 [02:18<10:05,  2.12it/s]
 17%|█▋        | 260/1545 [02:18<10:55,  1.96it/s]
                                                  
{'loss': 0.4419, 'grad_norm': 2.9558577807620168e-12, 'learning_rate': 8.317152103559872e-06, 'rewards/chosen': -17.75248146057129, 'rewards/rejected': -30.9813175201416, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 13.228837966918945, 'logps/chosen': -318.49920654296875, 'logps/rejected': -418.8551330566406, 'logits/chosen': -2.046048164367676, 'logits/rejected': -2.655272960662842, 'epoch': 0.17}

 17%|█▋        | 260/1545 [02:18<10:55,  1.96it/s]
 17%|█▋        | 261/1545 [02:19<11:08,  1.92it/s]
 17%|█▋        | 262/1545 [02:19<10:45,  1.99it/s]
 17%|█▋        | 263/1545 [02:20<11:14,  1.90it/s]
 17%|█▋        | 264/1545 [02:20<11:31,  1.85it/s]
 17%|█▋        | 265/1545 [02:21<11:34,  1.84it/s]
 17%|█▋        | 266/1545 [02:21<10:52,  1.96it/s]
 17%|█▋        | 267/1545 [02:22<11:28,  1.86it/s]
 17%|█▋        | 268/1545 [02:23<11:40,  1.82it/s]
 17%|█▋        | 269/1545 [02:23<11:10,  1.90it/s]
 17%|█▋        | 270/1545 [02:24<11:41,  1.82it/s]
                                                  
{'loss': 0.0115, 'grad_norm': 4.363059997558594e-05, 'learning_rate': 8.25242718446602e-06, 'rewards/chosen': -12.999127388000488, 'rewards/rejected': -31.61448097229004, 'rewards/accuracies': 1.0, 'rewards/margins': 18.61534881591797, 'logps/chosen': -296.9534606933594, 'logps/rejected': -435.48883056640625, 'logits/chosen': -1.737764596939087, 'logits/rejected': -3.4163742065429688, 'epoch': 0.17}

 17%|█▋        | 270/1545 [02:24<11:41,  1.82it/s]
 18%|█▊        | 271/1545 [02:24<12:05,  1.76it/s]
 18%|█▊        | 272/1545 [02:25<11:49,  1.79it/s]
 18%|█▊        | 273/1545 [02:25<11:08,  1.90it/s]
 18%|█▊        | 274/1545 [02:26<11:28,  1.85it/s]
 18%|█▊        | 275/1545 [02:26<10:19,  2.05it/s]
 18%|█▊        | 276/1545 [02:27<10:39,  1.99it/s]
 18%|█▊        | 277/1545 [02:27<10:04,  2.10it/s]
 18%|█▊        | 278/1545 [02:28<10:43,  1.97it/s]
 18%|█▊        | 279/1545 [02:28<10:56,  1.93it/s]
 18%|█▊        | 280/1545 [02:29<10:56,  1.93it/s]
                                                  
{'loss': 0.0759, 'grad_norm': 1.4028046280145645e-08, 'learning_rate': 8.18770226537217e-06, 'rewards/chosen': -28.01290512084961, 'rewards/rejected': -43.24602508544922, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 15.233118057250977, 'logps/chosen': -442.2579040527344, 'logps/rejected': -560.4046630859375, 'logits/chosen': -2.9247617721557617, 'logits/rejected': -3.853868007659912, 'epoch': 0.18}

 18%|█▊        | 280/1545 [02:29<10:56,  1.93it/s]
 18%|█▊        | 281/1545 [02:29<09:50,  2.14it/s]
 18%|█▊        | 282/1545 [02:30<10:36,  1.98it/s]
 18%|█▊        | 283/1545 [02:30<11:00,  1.91it/s]
 18%|█▊        | 284/1545 [02:31<10:33,  1.99it/s]
 18%|█▊        | 285/1545 [02:31<11:11,  1.88it/s]
 19%|█▊        | 286/1545 [02:32<11:30,  1.82it/s]
 19%|█▊        | 287/1545 [02:33<11:30,  1.82it/s]
 19%|█▊        | 288/1545 [02:33<10:37,  1.97it/s]
 19%|█▊        | 289/1545 [02:34<11:15,  1.86it/s]
 19%|█▉        | 290/1545 [02:34<11:23,  1.84it/s]
                                                  
{'loss': 3.3436, 'grad_norm': 5.0090140368830305e-17, 'learning_rate': 8.122977346278318e-06, 'rewards/chosen': -25.31455421447754, 'rewards/rejected': -46.47494888305664, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 21.16039276123047, 'logps/chosen': -403.2884216308594, 'logps/rejected': -592.1914672851562, 'logits/chosen': -2.8424019813537598, 'logits/rejected': -3.816692352294922, 'epoch': 0.19}

 19%|█▉        | 290/1545 [02:34<11:23,  1.84it/s]
 19%|█▉        | 291/1545 [02:35<11:07,  1.88it/s]
 19%|█▉        | 292/1545 [02:35<11:27,  1.82it/s]
 19%|█▉        | 293/1545 [02:36<11:42,  1.78it/s]
 19%|█▉        | 294/1545 [02:36<11:37,  1.79it/s]
 19%|█▉        | 295/1545 [02:37<10:51,  1.92it/s]
 19%|█▉        | 296/1545 [02:37<11:12,  1.86it/s]
 19%|█▉        | 297/1545 [02:39<16:07,  1.29it/s]
 19%|█▉        | 298/1545 [02:39<14:14,  1.46it/s]
 19%|█▉        | 299/1545 [02:40<13:15,  1.57it/s]
 19%|█▉        | 300/1545 [02:40<12:50,  1.62it/s]
                                                  
{'loss': 0.0003, 'grad_norm': 3.92901711165905e-09, 'learning_rate': 8.058252427184466e-06, 'rewards/chosen': -10.483701705932617, 'rewards/rejected': -42.29178237915039, 'rewards/accuracies': 1.0, 'rewards/margins': 31.808080673217773, 'logps/chosen': -243.2613983154297, 'logps/rejected': -532.6328735351562, 'logits/chosen': -1.451924443244934, 'logits/rejected': -4.144944190979004, 'epoch': 0.19}

 19%|█▉        | 300/1545 [02:40<12:50,  1.62it/s]
 19%|█▉        | 301/1545 [02:41<12:28,  1.66it/s]
 20%|█▉        | 302/1545 [02:41<11:27,  1.81it/s]
 20%|█▉        | 303/1545 [02:42<11:37,  1.78it/s]
 20%|█▉        | 304/1545 [02:42<11:35,  1.78it/s]
 20%|█▉        | 305/1545 [02:43<11:32,  1.79it/s]
 20%|█▉        | 306/1545 [02:44<11:30,  1.80it/s]
 20%|█▉        | 307/1545 [02:44<11:37,  1.78it/s]
 20%|█▉        | 308/1545 [02:45<11:41,  1.76it/s]
 20%|██        | 309/1545 [02:45<10:48,  1.91it/s]
 20%|██        | 310/1545 [02:46<11:14,  1.83it/s]
                                                  
{'loss': 0.4085, 'grad_norm': 520.0, 'learning_rate': 7.993527508090616e-06, 'rewards/chosen': -10.010625839233398, 'rewards/rejected': -25.10512924194336, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 15.094502449035645, 'logps/chosen': -236.5189971923828, 'logps/rejected': -370.3948059082031, 'logits/chosen': -2.0673410892486572, 'logits/rejected': -3.2709903717041016, 'epoch': 0.2}

 20%|██        | 310/1545 [02:46<11:14,  1.83it/s]
 20%|██        | 311/1545 [02:46<11:20,  1.81it/s]
 20%|██        | 312/1545 [02:47<11:16,  1.82it/s]
 20%|██        | 313/1545 [02:47<10:59,  1.87it/s]
 20%|██        | 314/1545 [02:48<11:09,  1.84it/s]
 20%|██        | 315/1545 [02:48<11:15,  1.82it/s]
 20%|██        | 316/1545 [02:49<10:30,  1.95it/s]
 21%|██        | 317/1545 [02:49<11:07,  1.84it/s]
 21%|██        | 318/1545 [02:50<11:17,  1.81it/s]
 21%|██        | 319/1545 [02:51<11:14,  1.82it/s]
 21%|██        | 320/1545 [02:51<11:03,  1.85it/s]
                                                  
{'loss': 0.0186, 'grad_norm': 348.0, 'learning_rate': 7.928802588996765e-06, 'rewards/chosen': -7.756754398345947, 'rewards/rejected': -28.2783260345459, 'rewards/accuracies': 1.0, 'rewards/margins': 20.52157211303711, 'logps/chosen': -252.49423217773438, 'logps/rejected': -395.2229919433594, 'logits/chosen': -1.2267696857452393, 'logits/rejected': -2.0097601413726807, 'epoch': 0.21}

 21%|██        | 320/1545 [02:51<11:03,  1.85it/s]
 21%|██        | 321/1545 [02:52<11:35,  1.76it/s]
 21%|██        | 322/1545 [02:52<10:21,  1.97it/s]
 21%|██        | 323/1545 [02:53<10:10,  2.00it/s]
 21%|██        | 324/1545 [02:53<10:31,  1.93it/s]
 21%|██        | 325/1545 [02:54<10:45,  1.89it/s]
 21%|██        | 326/1545 [02:54<11:08,  1.82it/s]
 21%|██        | 327/1545 [02:55<10:09,  2.00it/s]
 21%|██        | 328/1545 [02:55<10:38,  1.91it/s]
 21%|██▏       | 329/1545 [02:56<10:51,  1.87it/s]
 21%|██▏       | 330/1545 [02:56<10:48,  1.87it/s]
                                                  
{'loss': 0.6079, 'grad_norm': 3.0184188481996443e-16, 'learning_rate': 7.864077669902913e-06, 'rewards/chosen': -10.482062339782715, 'rewards/rejected': -27.2606201171875, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 16.7785587310791, 'logps/chosen': -251.0252227783203, 'logps/rejected': -369.49072265625, 'logits/chosen': -1.2209112644195557, 'logits/rejected': -2.5535261631011963, 'epoch': 0.21}

 21%|██▏       | 330/1545 [02:56<10:48,  1.87it/s]
 21%|██▏       | 331/1545 [02:57<10:39,  1.90it/s]
 21%|██▏       | 332/1545 [02:58<11:21,  1.78it/s]
 22%|██▏       | 333/1545 [02:58<11:12,  1.80it/s]
 22%|██▏       | 334/1545 [02:58<10:30,  1.92it/s]
 22%|██▏       | 335/1545 [02:59<10:59,  1.83it/s]
 22%|██▏       | 336/1545 [03:00<11:08,  1.81it/s]
 22%|██▏       | 337/1545 [03:00<11:01,  1.82it/s]
 22%|██▏       | 338/1545 [03:01<10:47,  1.86it/s]
 22%|██▏       | 339/1545 [03:01<11:00,  1.83it/s]
 22%|██▏       | 340/1545 [03:02<11:12,  1.79it/s]
                                                  
{'loss': 0.6174, 'grad_norm': 2.7755575615628914e-17, 'learning_rate': 7.799352750809061e-06, 'rewards/chosen': -15.824236869812012, 'rewards/rejected': -32.17787170410156, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 16.35363006591797, 'logps/chosen': -278.61468505859375, 'logps/rejected': -427.072021484375, 'logits/chosen': -2.014265537261963, 'logits/rejected': -2.9095723628997803, 'epoch': 0.22}

 22%|██▏       | 340/1545 [03:02<11:12,  1.79it/s]
 22%|██▏       | 341/1545 [03:02<10:38,  1.89it/s]
 22%|██▏       | 342/1545 [03:03<11:00,  1.82it/s]
 22%|██▏       | 343/1545 [03:03<09:57,  2.01it/s]
 22%|██▏       | 344/1545 [03:04<10:23,  1.93it/s]
 22%|██▏       | 345/1545 [03:04<09:48,  2.04it/s]
 22%|██▏       | 346/1545 [03:05<10:28,  1.91it/s]
 22%|██▏       | 347/1545 [03:05<10:40,  1.87it/s]
 23%|██▎       | 348/1545 [03:06<10:37,  1.88it/s]
 23%|██▎       | 349/1545 [03:07<10:33,  1.89it/s]
 23%|██▎       | 350/1545 [03:07<10:42,  1.86it/s]
                                                  
{'loss': 0.0, 'grad_norm': 4.607859233063394e-19, 'learning_rate': 7.734627831715211e-06, 'rewards/chosen': -10.83470344543457, 'rewards/rejected': -41.98002624511719, 'rewards/accuracies': 1.0, 'rewards/margins': 31.145328521728516, 'logps/chosen': -240.7436065673828, 'logps/rejected': -537.6058349609375, 'logits/chosen': -1.7035901546478271, 'logits/rejected': -3.3107447624206543, 'epoch': 0.23}

 23%|██▎       | 350/1545 [03:07<10:42,  1.86it/s]
 23%|██▎       | 351/1545 [03:08<10:46,  1.85it/s]
 23%|██▎       | 352/1545 [03:08<09:58,  1.99it/s]
 23%|██▎       | 353/1545 [03:09<10:20,  1.92it/s]
 23%|██▎       | 354/1545 [03:09<10:31,  1.89it/s]
 23%|██▎       | 355/1545 [03:10<10:33,  1.88it/s]
 23%|██▎       | 356/1545 [03:10<10:00,  1.98it/s]
 23%|██▎       | 357/1545 [03:11<10:30,  1.88it/s]
 23%|██▎       | 358/1545 [03:11<09:28,  2.09it/s]
 23%|██▎       | 359/1545 [03:12<09:52,  2.00it/s]
 23%|██▎       | 360/1545 [03:12<09:29,  2.08it/s]
                                                  
{'loss': 4.1093, 'grad_norm': 7.44648787076585e-12, 'learning_rate': 7.66990291262136e-06, 'rewards/chosen': -18.4683837890625, 'rewards/rejected': -34.16698455810547, 'rewards/accuracies': 0.699999988079071, 'rewards/margins': 15.698600769042969, 'logps/chosen': -337.5670166015625, 'logps/rejected': -447.79180908203125, 'logits/chosen': -2.0399553775787354, 'logits/rejected': -3.212721347808838, 'epoch': 0.23}

 23%|██▎       | 360/1545 [03:12<09:29,  2.08it/s]
 23%|██▎       | 361/1545 [03:13<10:10,  1.94it/s]
 23%|██▎       | 362/1545 [03:13<10:25,  1.89it/s]
 23%|██▎       | 363/1545 [03:14<10:11,  1.93it/s]
 24%|██▎       | 364/1545 [03:14<10:17,  1.91it/s]
 24%|██▎       | 365/1545 [03:15<10:34,  1.86it/s]
 24%|██▎       | 366/1545 [03:15<10:50,  1.81it/s]
 24%|██▍       | 367/1545 [03:16<10:09,  1.93it/s]
 24%|██▍       | 368/1545 [03:16<10:43,  1.83it/s]
 24%|██▍       | 369/1545 [03:17<10:49,  1.81it/s]
 24%|██▍       | 370/1545 [03:18<10:31,  1.86it/s]
                                                  
{'loss': 1.606, 'grad_norm': 0.011474609375, 'learning_rate': 7.605177993527508e-06, 'rewards/chosen': -15.731010437011719, 'rewards/rejected': -35.59105682373047, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 19.86004638671875, 'logps/chosen': -323.3865966796875, 'logps/rejected': -465.696044921875, 'logits/chosen': -2.1124587059020996, 'logits/rejected': -3.7612509727478027, 'epoch': 0.24}

 24%|██▍       | 370/1545 [03:18<10:31,  1.86it/s]
 24%|██▍       | 371/1545 [03:18<10:42,  1.83it/s]
 24%|██▍       | 372/1545 [03:19<10:48,  1.81it/s]
 24%|██▍       | 373/1545 [03:19<10:53,  1.79it/s]
 24%|██▍       | 374/1545 [03:20<10:04,  1.94it/s]
 24%|██▍       | 375/1545 [03:20<10:27,  1.86it/s]
 24%|██▍       | 376/1545 [03:21<10:43,  1.82it/s]
 24%|██▍       | 377/1545 [03:21<10:30,  1.85it/s]
 24%|██▍       | 378/1545 [03:22<10:32,  1.85it/s]
 25%|██▍       | 379/1545 [03:22<10:46,  1.80it/s]
 25%|██▍       | 380/1545 [03:23<09:41,  2.00it/s]
                                                  
{'loss': 0.0008, 'grad_norm': 2.7466739993542433e-10, 'learning_rate': 7.540453074433658e-06, 'rewards/chosen': -12.65916919708252, 'rewards/rejected': -49.6566047668457, 'rewards/accuracies': 1.0, 'rewards/margins': 36.99742889404297, 'logps/chosen': -260.13958740234375, 'logps/rejected': -610.1561279296875, 'logits/chosen': -1.6531693935394287, 'logits/rejected': -4.022229194641113, 'epoch': 0.25}

 25%|██▍       | 380/1545 [03:23<09:41,  2.00it/s]
 25%|██▍       | 381/1545 [03:23<09:32,  2.03it/s]
 25%|██▍       | 382/1545 [03:24<10:10,  1.90it/s]
 25%|██▍       | 383/1545 [03:24<10:24,  1.86it/s]
 25%|██▍       | 384/1545 [03:25<10:29,  1.85it/s]
 25%|██▍       | 385/1545 [03:25<09:47,  1.97it/s]
 25%|██▍       | 386/1545 [03:26<09:02,  2.14it/s]
 25%|██▌       | 387/1545 [03:26<09:38,  2.00it/s]
 25%|██▌       | 388/1545 [03:27<09:53,  1.95it/s]
 25%|██▌       | 389/1545 [03:27<09:30,  2.03it/s]
 25%|██▌       | 390/1545 [03:28<08:51,  2.17it/s]
                                                  
{'loss': 0.0693, 'grad_norm': 0.205078125, 'learning_rate': 7.475728155339807e-06, 'rewards/chosen': -22.143047332763672, 'rewards/rejected': -71.9369888305664, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 49.793941497802734, 'logps/chosen': -388.7550964355469, 'logps/rejected': -835.4134521484375, 'logits/chosen': -2.400635242462158, 'logits/rejected': -5.395668029785156, 'epoch': 0.25}

 25%|██▌       | 390/1545 [03:28<08:51,  2.17it/s]
 25%|██▌       | 391/1545 [03:28<08:24,  2.29it/s]
 25%|██▌       | 392/1545 [03:29<08:08,  2.36it/s]
 25%|██▌       | 393/1545 [03:29<08:41,  2.21it/s]
 26%|██▌       | 394/1545 [03:29<08:25,  2.28it/s]
 26%|██▌       | 395/1545 [03:30<09:14,  2.08it/s]
 26%|██▌       | 396/1545 [03:31<09:45,  1.96it/s]
 26%|██▌       | 397/1545 [03:31<09:43,  1.97it/s]
 26%|██▌       | 398/1545 [03:31<08:49,  2.16it/s]
 26%|██▌       | 399/1545 [03:32<09:31,  2.01it/s]
 26%|██▌       | 400/1545 [03:33<09:58,  1.91it/s]
                                                  
{'loss': 2.9874, 'grad_norm': 0.0002765655517578125, 'learning_rate': 7.411003236245955e-06, 'rewards/chosen': -18.571969985961914, 'rewards/rejected': -58.970985412597656, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 40.39901351928711, 'logps/chosen': -332.44952392578125, 'logps/rejected': -692.6171875, 'logits/chosen': -2.102853298187256, 'logits/rejected': -5.4109063148498535, 'epoch': 0.26}

 26%|██▌       | 400/1545 [03:33<09:58,  1.91it/s]
 26%|██▌       | 401/1545 [03:33<09:33,  1.99it/s]
 26%|██▌       | 402/1545 [03:34<10:00,  1.90it/s]
 26%|██▌       | 403/1545 [03:34<10:11,  1.87it/s]
 26%|██▌       | 404/1545 [03:35<10:16,  1.85it/s]
 26%|██▌       | 405/1545 [03:35<09:46,  1.94it/s]
 26%|██▋       | 406/1545 [03:36<10:07,  1.87it/s]
 26%|██▋       | 407/1545 [03:36<10:23,  1.83it/s]
 26%|██▋       | 408/1545 [03:37<10:05,  1.88it/s]
 26%|██▋       | 409/1545 [03:37<10:21,  1.83it/s]
 27%|██▋       | 410/1545 [03:38<10:25,  1.82it/s]
                                                  
{'loss': 0.0823, 'grad_norm': 1.1188966420050406e-16, 'learning_rate': 7.3462783171521046e-06, 'rewards/chosen': -16.90290069580078, 'rewards/rejected': -36.86577224731445, 'rewards/accuracies': 1.0, 'rewards/margins': 19.962865829467773, 'logps/chosen': -327.2436828613281, 'logps/rejected': -486.8968811035156, 'logits/chosen': -1.703176498413086, 'logits/rejected': -2.9482998847961426, 'epoch': 0.27}

 27%|██▋       | 410/1545 [03:38<10:25,  1.82it/s]
 27%|██▋       | 411/1545 [03:39<10:34,  1.79it/s]
 27%|██▋       | 412/1545 [03:40<13:59,  1.35it/s]
 27%|██▋       | 413/1545 [03:40<13:05,  1.44it/s]
 27%|██▋       | 414/1545 [03:41<12:11,  1.55it/s]
 27%|██▋       | 415/1545 [03:41<11:01,  1.71it/s]
 27%|██▋       | 416/1545 [03:42<11:09,  1.69it/s]
 27%|██▋       | 417/1545 [03:43<10:52,  1.73it/s]
 27%|██▋       | 418/1545 [03:43<10:42,  1.75it/s]
 27%|██▋       | 419/1545 [03:43<09:55,  1.89it/s]
 27%|██▋       | 420/1545 [03:44<10:21,  1.81it/s]
                                                  
{'loss': 1.0062, 'grad_norm': 0.0, 'learning_rate': 7.2815533980582534e-06, 'rewards/chosen': -29.327747344970703, 'rewards/rejected': -75.21330261230469, 'rewards/accuracies': 0.699999988079071, 'rewards/margins': 45.88555145263672, 'logps/chosen': -462.95111083984375, 'logps/rejected': -864.5911865234375, 'logits/chosen': -2.609936237335205, 'logits/rejected': -4.504573345184326, 'epoch': 0.27}

 27%|██▋       | 420/1545 [03:44<10:21,  1.81it/s]
 27%|██▋       | 421/1545 [03:45<10:26,  1.79it/s]
 27%|██▋       | 422/1545 [03:45<09:57,  1.88it/s]
 27%|██▋       | 423/1545 [03:46<10:24,  1.80it/s]
 27%|██▋       | 424/1545 [03:46<10:29,  1.78it/s]
 28%|██▊       | 425/1545 [03:47<10:22,  1.80it/s]
 28%|██▊       | 426/1545 [03:47<08:51,  2.11it/s]
 28%|██▊       | 427/1545 [03:48<09:37,  1.94it/s]
 28%|██▊       | 428/1545 [03:48<09:54,  1.88it/s]
 28%|██▊       | 429/1545 [03:49<09:56,  1.87it/s]
 28%|██▊       | 430/1545 [03:49<09:49,  1.89it/s]
                                                  
{'loss': 0.972, 'grad_norm': 2.5011104298755527e-12, 'learning_rate': 7.2168284789644015e-06, 'rewards/chosen': -48.82561492919922, 'rewards/rejected': -107.1943359375, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 58.36871337890625, 'logps/chosen': -632.4086303710938, 'logps/rejected': -1192.110107421875, 'logits/chosen': -3.8830108642578125, 'logits/rejected': -4.851853370666504, 'epoch': 0.28}

 28%|██▊       | 430/1545 [03:49<09:49,  1.89it/s]
 28%|██▊       | 431/1545 [03:50<10:13,  1.82it/s]
 28%|██▊       | 432/1545 [03:51<10:21,  1.79it/s]
 28%|██▊       | 433/1545 [03:51<09:41,  1.91it/s]
 28%|██▊       | 434/1545 [03:52<10:04,  1.84it/s]
 28%|██▊       | 435/1545 [03:52<10:13,  1.81it/s]
 28%|██▊       | 436/1545 [03:53<09:14,  2.00it/s]
 28%|██▊       | 437/1545 [03:53<08:51,  2.08it/s]
 28%|██▊       | 438/1545 [03:53<08:17,  2.22it/s]
 28%|██▊       | 439/1545 [03:54<08:56,  2.06it/s]
 28%|██▊       | 440/1545 [03:54<08:16,  2.22it/s]
                                                  
{'loss': 0.0, 'grad_norm': 6.054962083888163e-22, 'learning_rate': 7.152103559870551e-06, 'rewards/chosen': -39.671104431152344, 'rewards/rejected': -89.18429565429688, 'rewards/accuracies': 1.0, 'rewards/margins': 49.51319122314453, 'logps/chosen': -521.9129028320312, 'logps/rejected': -983.9114990234375, 'logits/chosen': -3.958955764770508, 'logits/rejected': -6.404815673828125, 'epoch': 0.28}

 28%|██▊       | 440/1545 [03:54<08:16,  2.22it/s]
 29%|██▊       | 441/1545 [03:55<08:23,  2.19it/s]
 29%|██▊       | 442/1545 [03:55<09:05,  2.02it/s]
 29%|██▊       | 443/1545 [03:56<08:28,  2.17it/s]
 29%|██▊       | 444/1545 [03:56<08:56,  2.05it/s]
 29%|██▉       | 445/1545 [03:57<08:45,  2.09it/s]
 29%|██▉       | 446/1545 [03:57<09:22,  1.95it/s]
 29%|██▉       | 447/1545 [03:58<09:39,  1.90it/s]
 29%|██▉       | 448/1545 [03:58<09:44,  1.88it/s]
 29%|██▉       | 449/1545 [03:59<09:15,  1.97it/s]
 29%|██▉       | 450/1545 [03:59<09:46,  1.87it/s]
                                                  
{'loss': 3.3081, 'grad_norm': 6.103515625e-05, 'learning_rate': 7.0873786407767e-06, 'rewards/chosen': -54.06614303588867, 'rewards/rejected': -93.74281311035156, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 39.676658630371094, 'logps/chosen': -693.2251586914062, 'logps/rejected': -1041.663818359375, 'logits/chosen': -3.7713088989257812, 'logits/rejected': -5.899226665496826, 'epoch': 0.29}

 29%|██▉       | 450/1545 [04:00<09:46,  1.87it/s]
 29%|██▉       | 451/1545 [04:00<10:07,  1.80it/s]
 29%|██▉       | 452/1545 [04:01<09:35,  1.90it/s]
 29%|██▉       | 453/1545 [04:01<09:48,  1.85it/s]
 29%|██▉       | 454/1545 [04:02<09:59,  1.82it/s]
 29%|██▉       | 455/1545 [04:02<10:01,  1.81it/s]
 30%|██▉       | 456/1545 [04:03<09:10,  1.98it/s]
 30%|██▉       | 457/1545 [04:03<08:34,  2.11it/s]
 30%|██▉       | 458/1545 [04:04<09:04,  2.00it/s]
 30%|██▉       | 459/1545 [04:04<09:22,  1.93it/s]
 30%|██▉       | 460/1545 [04:05<08:52,  2.04it/s]
                                                  
{'loss': 2.1395, 'grad_norm': 4.376943252282217e-12, 'learning_rate': 7.022653721682848e-06, 'rewards/chosen': -43.0927619934082, 'rewards/rejected': -76.77229309082031, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 33.679527282714844, 'logps/chosen': -589.8778686523438, 'logps/rejected': -876.2001953125, 'logits/chosen': -3.6679959297180176, 'logits/rejected': -4.975624084472656, 'epoch': 0.3}

 30%|██▉       | 460/1545 [04:05<08:52,  2.04it/s]
 30%|██▉       | 461/1545 [04:05<09:21,  1.93it/s]
 30%|██▉       | 462/1545 [04:06<09:31,  1.90it/s]
 30%|██▉       | 463/1545 [04:06<09:36,  1.88it/s]
 30%|███       | 464/1545 [04:07<09:31,  1.89it/s]
 30%|███       | 465/1545 [04:07<09:55,  1.81it/s]
 30%|███       | 466/1545 [04:08<10:06,  1.78it/s]
 30%|███       | 467/1545 [04:08<09:24,  1.91it/s]
 30%|███       | 468/1545 [04:09<09:52,  1.82it/s]
 30%|███       | 469/1545 [04:10<10:04,  1.78it/s]
 30%|███       | 470/1545 [04:10<09:50,  1.82it/s]
                                                  
{'loss': 0.0008, 'grad_norm': 0.0, 'learning_rate': 6.957928802588997e-06, 'rewards/chosen': -28.976566314697266, 'rewards/rejected': -84.09245300292969, 'rewards/accuracies': 1.0, 'rewards/margins': 55.11588668823242, 'logps/chosen': -440.51788330078125, 'logps/rejected': -952.6383056640625, 'logits/chosen': -2.842275619506836, 'logits/rejected': -4.066740989685059, 'epoch': 0.3}

 30%|███       | 470/1545 [04:10<09:50,  1.82it/s]
 30%|███       | 471/1545 [04:11<08:50,  2.02it/s]
 31%|███       | 472/1545 [04:11<09:23,  1.90it/s]
 31%|███       | 473/1545 [04:12<09:25,  1.90it/s]
 31%|███       | 474/1545 [04:12<09:18,  1.92it/s]
 31%|███       | 475/1545 [04:13<09:38,  1.85it/s]
 31%|███       | 476/1545 [04:13<09:45,  1.83it/s]
 31%|███       | 477/1545 [04:14<09:42,  1.83it/s]
 31%|███       | 478/1545 [04:14<08:53,  2.00it/s]
 31%|███       | 479/1545 [04:15<09:22,  1.90it/s]
 31%|███       | 480/1545 [04:15<09:25,  1.88it/s]
                                                  
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 6.893203883495147e-06, 'rewards/chosen': -20.838396072387695, 'rewards/rejected': -92.45664978027344, 'rewards/accuracies': 1.0, 'rewards/margins': 71.61825561523438, 'logps/chosen': -362.0668029785156, 'logps/rejected': -1039.4049072265625, 'logits/chosen': -2.1511878967285156, 'logits/rejected': -4.547484874725342, 'epoch': 0.31}

 31%|███       | 480/1545 [04:15<09:25,  1.88it/s]
 31%|███       | 481/1545 [04:16<09:25,  1.88it/s]
 31%|███       | 482/1545 [04:16<08:35,  2.06it/s]
 31%|███▏      | 483/1545 [04:17<09:04,  1.95it/s]
 31%|███▏      | 484/1545 [04:17<09:14,  1.91it/s]
 31%|███▏      | 485/1545 [04:18<08:58,  1.97it/s]
 31%|███▏      | 486/1545 [04:18<09:19,  1.89it/s]
 32%|███▏      | 487/1545 [04:19<09:37,  1.83it/s]
 32%|███▏      | 488/1545 [04:20<09:46,  1.80it/s]
 32%|███▏      | 489/1545 [04:20<09:03,  1.94it/s]
 32%|███▏      | 490/1545 [04:21<09:31,  1.85it/s]
                                                  
{'loss': 0.2412, 'grad_norm': 152.0, 'learning_rate': 6.828478964401295e-06, 'rewards/chosen': -30.904491424560547, 'rewards/rejected': -65.34832763671875, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 34.44383239746094, 'logps/chosen': -486.5516052246094, 'logps/rejected': -770.8928833007812, 'logits/chosen': -2.9180967807769775, 'logits/rejected': -4.752321720123291, 'epoch': 0.32}

 32%|███▏      | 490/1545 [04:21<09:31,  1.85it/s]
 32%|███▏      | 491/1545 [04:21<09:47,  1.79it/s]
 32%|███▏      | 492/1545 [04:22<09:18,  1.89it/s]
 32%|███▏      | 493/1545 [04:22<09:44,  1.80it/s]
 32%|███▏      | 494/1545 [04:23<09:47,  1.79it/s]
 32%|███▏      | 495/1545 [04:23<09:47,  1.79it/s]
 32%|███▏      | 496/1545 [04:24<09:05,  1.92it/s]
 32%|███▏      | 497/1545 [04:24<09:21,  1.86it/s]
 32%|███▏      | 498/1545 [04:25<09:39,  1.81it/s]
 32%|███▏      | 499/1545 [04:26<09:36,  1.81it/s]
 32%|███▏      | 500/1545 [04:26<09:58,  1.75it/s]
                                                  
{'loss': 1.302, 'grad_norm': 2.6072732907671432e-21, 'learning_rate': 6.763754045307444e-06, 'rewards/chosen': -14.026659965515137, 'rewards/rejected': -57.82221221923828, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 43.795555114746094, 'logps/chosen': -295.4187927246094, 'logps/rejected': -692.2587890625, 'logits/chosen': -1.8123325109481812, 'logits/rejected': -4.419227123260498, 'epoch': 0.32}

 32%|███▏      | 500/1545 [04:26<09:58,  1.75it/s]
 32%|███▏      | 501/1545 [04:27<10:10,  1.71it/s]
 32%|███▏      | 502/1545 [04:27<09:58,  1.74it/s]
 33%|███▎      | 503/1545 [04:28<09:47,  1.77it/s]
 33%|███▎      | 504/1545 [04:29<10:25,  1.66it/s]
 33%|███▎      | 505/1545 [04:29<10:18,  1.68it/s]
 33%|███▎      | 506/1545 [04:30<09:34,  1.81it/s]
 33%|███▎      | 507/1545 [04:30<09:55,  1.74it/s]
 33%|███▎      | 508/1545 [04:31<09:58,  1.73it/s]
 33%|███▎      | 509/1545 [04:31<09:36,  1.80it/s]
 33%|███▎      | 510/1545 [04:32<09:42,  1.78it/s]
                                                  
{'loss': 0.7018, 'grad_norm': 1.9063008949160576e-09, 'learning_rate': 6.6990291262135935e-06, 'rewards/chosen': -18.401165008544922, 'rewards/rejected': -41.45631408691406, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 23.05514907836914, 'logps/chosen': -362.13482666015625, 'logps/rejected': -531.1427612304688, 'logits/chosen': -1.9105132818222046, 'logits/rejected': -3.823967695236206, 'epoch': 0.33}

 33%|███▎      | 510/1545 [04:32<09:42,  1.78it/s]
 33%|███▎      | 511/1545 [04:33<09:52,  1.75it/s]
 33%|███▎      | 512/1545 [04:33<09:40,  1.78it/s]
 33%|███▎      | 513/1545 [04:33<09:00,  1.91it/s]
 33%|███▎      | 514/1545 [04:34<09:16,  1.85it/s]
 33%|███▎      | 515/1545 [04:35<09:22,  1.83it/s]
 33%|███▎      | 516/1545 [04:35<09:13,  1.86it/s]
 33%|███▎      | 517/1545 [04:36<09:30,  1.80it/s]
 34%|███▎      | 518/1545 [04:36<09:33,  1.79it/s]
 34%|███▎      | 519/1545 [04:37<09:31,  1.80it/s]
 34%|███▎      | 520/1545 [04:37<08:48,  1.94it/s]
                                                  
{'loss': 0.9755, 'grad_norm': 3.924811864397526e-17, 'learning_rate': 6.6343042071197415e-06, 'rewards/chosen': -7.66559362411499, 'rewards/rejected': -35.98261642456055, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 28.317020416259766, 'logps/chosen': -237.4998321533203, 'logps/rejected': -480.2190856933594, 'logits/chosen': -1.257673978805542, 'logits/rejected': -3.1603283882141113, 'epoch': 0.34}

 34%|███▎      | 520/1545 [04:37<08:48,  1.94it/s]
 34%|███▎      | 521/1545 [04:38<09:07,  1.87it/s]
 34%|███▍      | 522/1545 [04:38<09:11,  1.85it/s]
 34%|███▍      | 523/1545 [04:39<09:10,  1.86it/s]
 34%|███▍      | 524/1545 [04:39<09:03,  1.88it/s]
 34%|███▍      | 525/1545 [04:40<09:23,  1.81it/s]
 34%|███▍      | 526/1545 [04:41<13:50,  1.23it/s]
 34%|███▍      | 527/1545 [04:42<11:38,  1.46it/s]
 34%|███▍      | 528/1545 [04:42<10:59,  1.54it/s]
 34%|███▍      | 529/1545 [04:43<10:37,  1.59it/s]
 34%|███▍      | 530/1545 [04:44<10:09,  1.67it/s]
                                                  
{'loss': 0.4228, 'grad_norm': 0.0, 'learning_rate': 6.56957928802589e-06, 'rewards/chosen': -30.713363647460938, 'rewards/rejected': -60.528602600097656, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 29.81524085998535, 'logps/chosen': -450.9524841308594, 'logps/rejected': -714.2261962890625, 'logits/chosen': -3.075618267059326, 'logits/rejected': -4.854428768157959, 'epoch': 0.34}

 34%|███▍      | 530/1545 [04:44<10:09,  1.67it/s]
 34%|███▍      | 531/1545 [04:44<09:39,  1.75it/s]
 34%|███▍      | 532/1545 [04:45<09:42,  1.74it/s]
 34%|███▍      | 533/1545 [04:45<09:37,  1.75it/s]
 35%|███▍      | 534/1545 [04:46<08:54,  1.89it/s]
 35%|███▍      | 535/1545 [04:46<09:14,  1.82it/s]
 35%|███▍      | 536/1545 [04:47<09:17,  1.81it/s]
 35%|███▍      | 537/1545 [04:47<09:14,  1.82it/s]
 35%|███▍      | 538/1545 [04:48<09:06,  1.84it/s]
 35%|███▍      | 539/1545 [04:48<09:18,  1.80it/s]
 35%|███▍      | 540/1545 [04:49<09:20,  1.79it/s]
                                                  
{'loss': 0.723, 'grad_norm': 4.440892098500626e-15, 'learning_rate': 6.50485436893204e-06, 'rewards/chosen': -16.0915584564209, 'rewards/rejected': -53.74153518676758, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 37.64997482299805, 'logps/chosen': -319.9918518066406, 'logps/rejected': -653.3815307617188, 'logits/chosen': -1.6358880996704102, 'logits/rejected': -3.925060749053955, 'epoch': 0.35}

 35%|███▍      | 540/1545 [04:49<09:20,  1.79it/s]
 35%|███▌      | 541/1545 [04:49<08:44,  1.91it/s]
 35%|███▌      | 542/1545 [04:50<09:08,  1.83it/s]
 35%|███▌      | 543/1545 [04:51<09:12,  1.81it/s]
 35%|███▌      | 544/1545 [04:51<09:09,  1.82it/s]
 35%|███▌      | 545/1545 [04:52<09:01,  1.85it/s]
 35%|███▌      | 546/1545 [04:52<09:09,  1.82it/s]
 35%|███▌      | 547/1545 [04:53<09:15,  1.80it/s]
 35%|███▌      | 548/1545 [04:53<08:38,  1.92it/s]
 36%|███▌      | 549/1545 [04:54<08:53,  1.87it/s]
 36%|███▌      | 550/1545 [04:54<08:57,  1.85it/s]
                                                  
{'loss': 0.0105, 'grad_norm': 3.9257486150745535e-13, 'learning_rate': 6.440129449838188e-06, 'rewards/chosen': -14.184236526489258, 'rewards/rejected': -45.90851593017578, 'rewards/accuracies': 1.0, 'rewards/margins': 31.72427749633789, 'logps/chosen': -261.56756591796875, 'logps/rejected': -570.337890625, 'logits/chosen': -1.7183490991592407, 'logits/rejected': -3.5872623920440674, 'epoch': 0.36}

 36%|███▌      | 550/1545 [04:54<08:57,  1.85it/s]
 36%|███▌      | 551/1545 [04:55<09:07,  1.81it/s]
 36%|███▌      | 552/1545 [04:55<08:51,  1.87it/s]
 36%|███▌      | 553/1545 [04:56<09:04,  1.82it/s]
 36%|███▌      | 554/1545 [04:57<09:02,  1.83it/s]
 36%|███▌      | 555/1545 [04:57<08:34,  1.92it/s]
 36%|███▌      | 556/1545 [04:58<08:50,  1.86it/s]
 36%|███▌      | 557/1545 [04:58<08:55,  1.84it/s]
 36%|███▌      | 558/1545 [04:59<09:02,  1.82it/s]
 36%|███▌      | 559/1545 [04:59<08:16,  1.99it/s]
 36%|███▌      | 560/1545 [04:59<07:39,  2.15it/s]
                                                  
{'loss': 1.7896, 'grad_norm': 7.104873657226562e-05, 'learning_rate': 6.375404530744337e-06, 'rewards/chosen': -30.25199317932129, 'rewards/rejected': -56.004981994628906, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 25.75299072265625, 'logps/chosen': -435.36322021484375, 'logps/rejected': -665.7625732421875, 'logits/chosen': -1.7875016927719116, 'logits/rejected': -3.8524105548858643, 'epoch': 0.36}

 36%|███▌      | 560/1545 [05:00<07:39,  2.15it/s]
 36%|███▋      | 561/1545 [05:00<08:08,  2.01it/s]
 36%|███▋      | 562/1545 [05:01<08:29,  1.93it/s]
 36%|███▋      | 563/1545 [05:01<07:58,  2.05it/s]
 37%|███▋      | 564/1545 [05:02<08:22,  1.95it/s]
 37%|███▋      | 565/1545 [05:02<08:33,  1.91it/s]
 37%|███▋      | 566/1545 [05:03<08:37,  1.89it/s]
 37%|███▋      | 567/1545 [05:03<08:25,  1.94it/s]
 37%|███▋      | 568/1545 [05:04<08:39,  1.88it/s]
 37%|███▋      | 569/1545 [05:04<08:46,  1.85it/s]
 37%|███▋      | 570/1545 [05:05<08:16,  1.96it/s]
                                                  
{'loss': 0.0057, 'grad_norm': 1.0302869668521453e-12, 'learning_rate': 6.310679611650487e-06, 'rewards/chosen': -9.948836326599121, 'rewards/rejected': -50.712440490722656, 'rewards/accuracies': 1.0, 'rewards/margins': 40.76360321044922, 'logps/chosen': -243.68978881835938, 'logps/rejected': -616.48486328125, 'logits/chosen': -1.0023655891418457, 'logits/rejected': -3.4231104850769043, 'epoch': 0.37}

 37%|███▋      | 570/1545 [05:05<08:16,  1.96it/s]
 37%|███▋      | 571/1545 [05:05<08:46,  1.85it/s]
 37%|███▋      | 572/1545 [05:06<08:48,  1.84it/s]
 37%|███▋      | 573/1545 [05:06<08:45,  1.85it/s]
 37%|███▋      | 574/1545 [05:07<08:32,  1.90it/s]
 37%|███▋      | 575/1545 [05:08<08:43,  1.85it/s]
 37%|███▋      | 576/1545 [05:08<08:45,  1.84it/s]
 37%|███▋      | 577/1545 [05:09<08:19,  1.94it/s]
 37%|███▋      | 578/1545 [05:09<08:36,  1.87it/s]
 37%|███▋      | 579/1545 [05:10<08:48,  1.83it/s]
 38%|███▊      | 580/1545 [05:10<08:46,  1.83it/s]
                                                  
{'loss': 0.0143, 'grad_norm': 0.0, 'learning_rate': 6.245954692556635e-06, 'rewards/chosen': -18.19172477722168, 'rewards/rejected': -61.596435546875, 'rewards/accuracies': 1.0, 'rewards/margins': 43.40471267700195, 'logps/chosen': -331.9469909667969, 'logps/rejected': -743.6862182617188, 'logits/chosen': -1.4042354822158813, 'logits/rejected': -3.8643798828125, 'epoch': 0.38}

 38%|███▊      | 580/1545 [05:10<08:46,  1.83it/s]
 38%|███▊      | 581/1545 [05:11<08:20,  1.93it/s]
 38%|███▊      | 582/1545 [05:11<08:42,  1.84it/s]
 38%|███▊      | 583/1545 [05:12<08:53,  1.80it/s]
 38%|███▊      | 584/1545 [05:12<08:40,  1.85it/s]
 38%|███▊      | 585/1545 [05:13<08:53,  1.80it/s]
 38%|███▊      | 586/1545 [05:14<09:09,  1.74it/s]
 38%|███▊      | 587/1545 [05:14<09:00,  1.77it/s]
 38%|███▊      | 588/1545 [05:15<08:39,  1.84it/s]
 38%|███▊      | 589/1545 [05:15<08:46,  1.82it/s]
 38%|███▊      | 590/1545 [05:16<08:40,  1.83it/s]
                                                  
{'loss': 0.0179, 'grad_norm': 77.5, 'learning_rate': 6.181229773462784e-06, 'rewards/chosen': -18.66933822631836, 'rewards/rejected': -50.0496826171875, 'rewards/accuracies': 1.0, 'rewards/margins': 31.380340576171875, 'logps/chosen': -321.9429626464844, 'logps/rejected': -626.2196655273438, 'logits/chosen': -1.854098916053772, 'logits/rejected': -3.260258436203003, 'epoch': 0.38}

 38%|███▊      | 590/1545 [05:16<08:40,  1.83it/s]
 38%|███▊      | 591/1545 [05:16<08:12,  1.94it/s]
 38%|███▊      | 592/1545 [05:17<08:37,  1.84it/s]
 38%|███▊      | 593/1545 [05:17<08:51,  1.79it/s]
 38%|███▊      | 594/1545 [05:18<07:55,  2.00it/s]
 39%|███▊      | 595/1545 [05:18<07:44,  2.05it/s]
 39%|███▊      | 596/1545 [05:19<08:24,  1.88it/s]
 39%|███▊      | 597/1545 [05:19<08:37,  1.83it/s]
 39%|███▊      | 598/1545 [05:20<07:48,  2.02it/s]
 39%|███▉      | 599/1545 [05:20<07:27,  2.11it/s]
 39%|███▉      | 600/1545 [05:21<07:53,  2.00it/s]
                                                  
{'loss': 0.0001, 'grad_norm': 0.5234375, 'learning_rate': 6.116504854368932e-06, 'rewards/chosen': -20.77777099609375, 'rewards/rejected': -52.45641326904297, 'rewards/accuracies': 1.0, 'rewards/margins': 31.678646087646484, 'logps/chosen': -362.299072265625, 'logps/rejected': -631.9798583984375, 'logits/chosen': -1.6714366674423218, 'logits/rejected': -3.967179775238037, 'epoch': 0.39}

 39%|███▉      | 600/1545 [05:21<07:53,  2.00it/s]
 39%|███▉      | 601/1545 [05:21<08:15,  1.90it/s]
 39%|███▉      | 602/1545 [05:22<08:17,  1.90it/s]
 39%|███▉      | 603/1545 [05:22<08:08,  1.93it/s]
 39%|███▉      | 604/1545 [05:23<08:23,  1.87it/s]
 39%|███▉      | 605/1545 [05:24<08:28,  1.85it/s]
 39%|███▉      | 606/1545 [05:24<08:07,  1.92it/s]
 39%|███▉      | 607/1545 [05:24<07:28,  2.09it/s]
 39%|███▉      | 608/1545 [05:25<07:53,  1.98it/s]
 39%|███▉      | 609/1545 [05:25<08:07,  1.92it/s]
 39%|███▉      | 610/1545 [05:26<07:55,  1.97it/s]
                                                  
{'loss': 0.0579, 'grad_norm': 0.0, 'learning_rate': 6.0517799352750815e-06, 'rewards/chosen': -18.156063079833984, 'rewards/rejected': -48.84654998779297, 'rewards/accuracies': 1.0, 'rewards/margins': 30.690486907958984, 'logps/chosen': -305.2608947753906, 'logps/rejected': -596.3084716796875, 'logits/chosen': -1.5014350414276123, 'logits/rejected': -3.3216071128845215, 'epoch': 0.39}

 39%|███▉      | 610/1545 [05:26<07:55,  1.97it/s]
 40%|███▉      | 611/1545 [05:27<08:20,  1.86it/s]
 40%|███▉      | 612/1545 [05:27<07:32,  2.06it/s]
 40%|███▉      | 613/1545 [05:27<07:55,  1.96it/s]
 40%|███▉      | 614/1545 [05:28<07:28,  2.08it/s]
 40%|███▉      | 615/1545 [05:28<07:56,  1.95it/s]
 40%|███▉      | 616/1545 [05:29<08:07,  1.91it/s]
 40%|███▉      | 617/1545 [05:30<08:09,  1.90it/s]
 40%|████      | 618/1545 [05:30<08:01,  1.92it/s]
 40%|████      | 619/1545 [05:30<07:22,  2.09it/s]
 40%|████      | 620/1545 [05:31<07:43,  1.99it/s]
                                                  
{'loss': 0.0011, 'grad_norm': 12.3125, 'learning_rate': 5.9870550161812304e-06, 'rewards/chosen': -20.277729034423828, 'rewards/rejected': -44.97526931762695, 'rewards/accuracies': 1.0, 'rewards/margins': 24.69754409790039, 'logps/chosen': -371.6828308105469, 'logps/rejected': -576.7119750976562, 'logits/chosen': -1.2356998920440674, 'logits/rejected': -2.410062074661255, 'epoch': 0.4}

 40%|████      | 620/1545 [05:31<07:43,  1.99it/s]
 40%|████      | 621/1545 [05:32<07:59,  1.93it/s]
 40%|████      | 622/1545 [05:32<08:12,  1.88it/s]
 40%|████      | 623/1545 [05:33<08:25,  1.82it/s]
 40%|████      | 624/1545 [05:33<08:24,  1.83it/s]
 40%|████      | 625/1545 [05:34<07:55,  1.94it/s]
 41%|████      | 626/1545 [05:34<08:20,  1.84it/s]
 41%|████      | 627/1545 [05:35<08:27,  1.81it/s]
 41%|████      | 628/1545 [05:35<08:07,  1.88it/s]
 41%|████      | 629/1545 [05:36<08:24,  1.81it/s]
 41%|████      | 630/1545 [05:37<08:29,  1.80it/s]
                                                  
{'loss': 0.0001, 'grad_norm': 6.606569513678551e-09, 'learning_rate': 5.9223300970873785e-06, 'rewards/chosen': -22.200729370117188, 'rewards/rejected': -52.896087646484375, 'rewards/accuracies': 1.0, 'rewards/margins': 30.695358276367188, 'logps/chosen': -359.304931640625, 'logps/rejected': -632.1669921875, 'logits/chosen': -2.2159793376922607, 'logits/rejected': -4.487706184387207, 'epoch': 0.41}

 41%|████      | 630/1545 [05:37<08:29,  1.80it/s]
 41%|████      | 631/1545 [05:37<08:32,  1.78it/s]
 41%|████      | 632/1545 [05:38<07:54,  1.92it/s]
 41%|████      | 633/1545 [05:38<08:11,  1.86it/s]
 41%|████      | 634/1545 [05:39<08:16,  1.83it/s]
 41%|████      | 635/1545 [05:39<08:01,  1.89it/s]
 41%|████      | 636/1545 [05:40<07:07,  2.13it/s]
 41%|████      | 637/1545 [05:40<07:35,  1.99it/s]
 41%|████▏     | 638/1545 [05:41<07:53,  1.92it/s]
 41%|████▏     | 639/1545 [05:41<07:36,  1.98it/s]
 41%|████▏     | 640/1545 [05:42<07:54,  1.91it/s]
                                                  
{'loss': 0.0756, 'grad_norm': 2.8731357570865868e-18, 'learning_rate': 5.857605177993528e-06, 'rewards/chosen': -37.544307708740234, 'rewards/rejected': -75.83995056152344, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 38.295654296875, 'logps/chosen': -509.95721435546875, 'logps/rejected': -876.0435791015625, 'logits/chosen': -3.4378981590270996, 'logits/rejected': -4.655713081359863, 'epoch': 0.41}

 41%|████▏     | 640/1545 [05:42<07:54,  1.91it/s]
 41%|████▏     | 641/1545 [05:43<11:29,  1.31it/s]
 42%|████▏     | 642/1545 [05:44<10:22,  1.45it/s]
 42%|████▏     | 643/1545 [05:44<09:02,  1.66it/s]
 42%|████▏     | 644/1545 [05:45<08:56,  1.68it/s]
 42%|████▏     | 645/1545 [05:45<08:41,  1.73it/s]
 42%|████▏     | 646/1545 [05:46<08:27,  1.77it/s]
 42%|████▏     | 647/1545 [05:46<08:11,  1.83it/s]
 42%|████▏     | 648/1545 [05:47<08:16,  1.81it/s]
 42%|████▏     | 649/1545 [05:47<08:16,  1.81it/s]
 42%|████▏     | 650/1545 [05:48<07:40,  1.94it/s]
                                                  
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 5.792880258899677e-06, 'rewards/chosen': -17.858400344848633, 'rewards/rejected': -68.02870178222656, 'rewards/accuracies': 1.0, 'rewards/margins': 50.1702995300293, 'logps/chosen': -305.210205078125, 'logps/rejected': -778.7840576171875, 'logits/chosen': -2.0918753147125244, 'logits/rejected': -5.265947341918945, 'epoch': 0.42}

 42%|████▏     | 650/1545 [05:48<07:40,  1.94it/s]
 42%|████▏     | 651/1545 [05:48<07:57,  1.87it/s]
 42%|████▏     | 652/1545 [05:49<08:05,  1.84it/s]
 42%|████▏     | 653/1545 [05:49<07:19,  2.03it/s]
 42%|████▏     | 654/1545 [05:50<07:01,  2.11it/s]
 42%|████▏     | 655/1545 [05:50<07:26,  1.99it/s]
 42%|████▏     | 656/1545 [05:51<07:46,  1.91it/s]
 43%|████▎     | 657/1545 [05:51<08:01,  1.84it/s]
 43%|████▎     | 658/1545 [05:52<07:45,  1.91it/s]
 43%|████▎     | 659/1545 [05:52<07:58,  1.85it/s]
 43%|████▎     | 660/1545 [05:53<07:58,  1.85it/s]
                                                  
{'loss': 2.3511, 'grad_norm': 0.0, 'learning_rate': 5.728155339805825e-06, 'rewards/chosen': -37.305686950683594, 'rewards/rejected': -80.12673950195312, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 42.821044921875, 'logps/chosen': -520.0347290039062, 'logps/rejected': -906.5696411132812, 'logits/chosen': -3.2962348461151123, 'logits/rejected': -6.164005279541016, 'epoch': 0.43}

 43%|████▎     | 660/1545 [05:53<07:58,  1.85it/s]
 43%|████▎     | 661/1545 [05:53<07:34,  1.95it/s]
 43%|████▎     | 662/1545 [05:54<07:50,  1.88it/s]
 43%|████▎     | 663/1545 [05:55<08:01,  1.83it/s]
 43%|████▎     | 664/1545 [05:55<08:02,  1.83it/s]
 43%|████▎     | 665/1545 [05:55<07:23,  1.98it/s]
 43%|████▎     | 666/1545 [05:56<07:41,  1.90it/s]
 43%|████▎     | 667/1545 [05:56<06:58,  2.10it/s]
 43%|████▎     | 668/1545 [05:57<07:18,  2.00it/s]
 43%|████▎     | 669/1545 [05:57<06:56,  2.10it/s]
 43%|████▎     | 670/1545 [05:58<07:20,  1.99it/s]
                                                  
{'loss': 0.0, 'grad_norm': 0.00046539306640625, 'learning_rate': 5.663430420711975e-06, 'rewards/chosen': -21.083389282226562, 'rewards/rejected': -60.96876907348633, 'rewards/accuracies': 1.0, 'rewards/margins': 39.885379791259766, 'logps/chosen': -365.4623107910156, 'logps/rejected': -721.8714599609375, 'logits/chosen': -2.13993501663208, 'logits/rejected': -5.14273738861084, 'epoch': 0.43}

 43%|████▎     | 670/1545 [05:58<07:20,  1.99it/s]
 43%|████▎     | 671/1545 [05:59<07:44,  1.88it/s]
 43%|████▎     | 672/1545 [05:59<07:41,  1.89it/s]
 44%|████▎     | 673/1545 [06:00<07:37,  1.91it/s]
 44%|████▎     | 674/1545 [06:00<07:50,  1.85it/s]
 44%|████▎     | 675/1545 [06:01<07:52,  1.84it/s]
 44%|████▍     | 676/1545 [06:01<07:21,  1.97it/s]
 44%|████▍     | 677/1545 [06:02<07:38,  1.89it/s]
 44%|████▍     | 678/1545 [06:02<07:46,  1.86it/s]
 44%|████▍     | 679/1545 [06:03<07:39,  1.88it/s]
 44%|████▍     | 680/1545 [06:03<07:04,  2.04it/s]
                                                  
{'loss': 0.001, 'grad_norm': 9.492850949754938e-12, 'learning_rate': 5.598705501618124e-06, 'rewards/chosen': -25.24799346923828, 'rewards/rejected': -60.609230041503906, 'rewards/accuracies': 1.0, 'rewards/margins': 35.361228942871094, 'logps/chosen': -402.2320251464844, 'logps/rejected': -727.8626708984375, 'logits/chosen': -2.94899845123291, 'logits/rejected': -5.123431205749512, 'epoch': 0.44}

 44%|████▍     | 680/1545 [06:03<07:04,  2.04it/s]
 44%|████▍     | 681/1545 [06:04<07:28,  1.93it/s]
 44%|████▍     | 682/1545 [06:04<07:33,  1.90it/s]
 44%|████▍     | 683/1545 [06:05<07:32,  1.91it/s]
 44%|████▍     | 684/1545 [06:05<06:27,  2.22it/s]
 44%|████▍     | 685/1545 [06:06<07:03,  2.03it/s]
 44%|████▍     | 686/1545 [06:06<06:33,  2.19it/s]
 44%|████▍     | 687/1545 [06:07<06:58,  2.05it/s]
 45%|████▍     | 688/1545 [06:07<06:43,  2.12it/s]
 45%|████▍     | 689/1545 [06:08<07:11,  1.98it/s]
 45%|████▍     | 690/1545 [06:08<07:25,  1.92it/s]
                                                  
{'loss': 0.2533, 'grad_norm': 4.76837158203125e-07, 'learning_rate': 5.533980582524272e-06, 'rewards/chosen': -26.505752563476562, 'rewards/rejected': -50.40888595581055, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 23.90313720703125, 'logps/chosen': -408.7076721191406, 'logps/rejected': -599.2103881835938, 'logits/chosen': -2.7856857776641846, 'logits/rejected': -5.5106892585754395, 'epoch': 0.45}

 45%|████▍     | 690/1545 [06:08<07:25,  1.92it/s]
 45%|████▍     | 691/1545 [06:09<07:36,  1.87it/s]
 45%|████▍     | 692/1545 [06:09<06:37,  2.15it/s]
 45%|████▍     | 693/1545 [06:09<05:52,  2.42it/s]
 45%|████▍     | 694/1545 [06:10<06:28,  2.19it/s]
 45%|████▍     | 695/1545 [06:10<06:50,  2.07it/s]
 45%|████▌     | 696/1545 [06:11<06:01,  2.35it/s]
 45%|████▌     | 697/1545 [06:11<05:24,  2.61it/s]
 45%|████▌     | 698/1545 [06:11<05:33,  2.54it/s]
 45%|████▌     | 699/1545 [06:12<05:48,  2.43it/s]
 45%|████▌     | 700/1545 [06:12<05:51,  2.40it/s]
                                                  
{'loss': 0.3027, 'grad_norm': 0.0, 'learning_rate': 5.4692556634304216e-06, 'rewards/chosen': -22.45337677001953, 'rewards/rejected': -68.20580291748047, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 45.752418518066406, 'logps/chosen': -372.1856384277344, 'logps/rejected': -798.4400634765625, 'logits/chosen': -2.2510809898376465, 'logits/rejected': -5.148565292358398, 'epoch': 0.45}

 45%|████▌     | 700/1545 [06:12<05:51,  2.40it/s]
 45%|████▌     | 701/1545 [06:13<06:09,  2.28it/s]
 45%|████▌     | 702/1545 [06:13<06:19,  2.22it/s]
 46%|████▌     | 703/1545 [06:14<06:16,  2.24it/s]
 46%|████▌     | 704/1545 [06:14<06:43,  2.08it/s]
 46%|████▌     | 705/1545 [06:15<07:11,  1.95it/s]
 46%|████▌     | 706/1545 [06:15<06:52,  2.03it/s]
 46%|████▌     | 707/1545 [06:16<07:00,  1.99it/s]
 46%|████▌     | 708/1545 [06:16<06:29,  2.15it/s]
 46%|████▌     | 709/1545 [06:17<06:29,  2.15it/s]
 46%|████▌     | 710/1545 [06:17<06:47,  2.05it/s]
                                                  
{'loss': 0.5746, 'grad_norm': 0.0, 'learning_rate': 5.4045307443365705e-06, 'rewards/chosen': -27.447168350219727, 'rewards/rejected': -84.84578704833984, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 57.39862060546875, 'logps/chosen': -426.19427490234375, 'logps/rejected': -969.3411865234375, 'logits/chosen': -2.232661724090576, 'logits/rejected': -4.795763969421387, 'epoch': 0.46}

 46%|████▌     | 710/1545 [06:17<06:47,  2.05it/s]
 46%|████▌     | 711/1545 [06:18<07:15,  1.92it/s]
 46%|████▌     | 712/1545 [06:18<07:04,  1.96it/s]
 46%|████▌     | 713/1545 [06:19<07:04,  1.96it/s]
 46%|████▌     | 714/1545 [06:19<07:20,  1.89it/s]
 46%|████▋     | 715/1545 [06:20<07:16,  1.90it/s]
 46%|████▋     | 716/1545 [06:20<07:07,  1.94it/s]
 46%|████▋     | 717/1545 [06:21<07:12,  1.91it/s]
 46%|████▋     | 718/1545 [06:22<07:29,  1.84it/s]
 47%|████▋     | 719/1545 [06:22<07:15,  1.90it/s]
 47%|████▋     | 720/1545 [06:23<07:19,  1.88it/s]
                                                  
{'loss': 0.003, 'grad_norm': 0.0, 'learning_rate': 5.3398058252427185e-06, 'rewards/chosen': -38.11956787109375, 'rewards/rejected': -97.89371490478516, 'rewards/accuracies': 1.0, 'rewards/margins': 59.774139404296875, 'logps/chosen': -570.1392822265625, 'logps/rejected': -1108.6182861328125, 'logits/chosen': -2.4496023654937744, 'logits/rejected': -5.000131607055664, 'epoch': 0.47}

 47%|████▋     | 720/1545 [06:23<07:19,  1.88it/s]
 47%|████▋     | 721/1545 [06:23<07:25,  1.85it/s]
 47%|████▋     | 722/1545 [06:24<07:37,  1.80it/s]
 47%|████▋     | 723/1545 [06:24<07:19,  1.87it/s]
 47%|████▋     | 724/1545 [06:25<07:21,  1.86it/s]
 47%|████▋     | 725/1545 [06:25<07:28,  1.83it/s]
 47%|████▋     | 726/1545 [06:26<07:26,  1.83it/s]
 47%|████▋     | 727/1545 [06:26<07:04,  1.93it/s]
 47%|████▋     | 728/1545 [06:27<07:10,  1.90it/s]
 47%|████▋     | 729/1545 [06:27<07:22,  1.84it/s]
 47%|████▋     | 730/1545 [06:28<07:00,  1.94it/s]
                                                  
{'loss': 1.7073, 'grad_norm': 2.3418766925686896e-16, 'learning_rate': 5.275080906148867e-06, 'rewards/chosen': -44.976646423339844, 'rewards/rejected': -86.87692260742188, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 41.9002799987793, 'logps/chosen': -594.9097290039062, 'logps/rejected': -975.1007690429688, 'logits/chosen': -3.272984027862549, 'logits/rejected': -5.4031596183776855, 'epoch': 0.47}

 47%|████▋     | 730/1545 [06:28<07:00,  1.94it/s]
 47%|████▋     | 731/1545 [06:28<07:05,  1.91it/s]
 47%|████▋     | 732/1545 [06:29<07:16,  1.86it/s]
 47%|████▋     | 733/1545 [06:29<06:59,  1.93it/s]
 48%|████▊     | 734/1545 [06:30<07:06,  1.90it/s]
 48%|████▊     | 735/1545 [06:31<07:10,  1.88it/s]
 48%|████▊     | 736/1545 [06:31<07:15,  1.86it/s]
 48%|████▊     | 737/1545 [06:32<06:55,  1.94it/s]
 48%|████▊     | 738/1545 [06:32<07:00,  1.92it/s]
 48%|████▊     | 739/1545 [06:33<07:10,  1.87it/s]
 48%|████▊     | 740/1545 [06:33<06:53,  1.95it/s]
                                                  
{'loss': 0.0581, 'grad_norm': 30.5, 'learning_rate': 5.210355987055017e-06, 'rewards/chosen': -43.055625915527344, 'rewards/rejected': -78.43653106689453, 'rewards/accuracies': 1.0, 'rewards/margins': 35.38090133666992, 'logps/chosen': -582.3439331054688, 'logps/rejected': -901.49658203125, 'logits/chosen': -3.8236382007598877, 'logits/rejected': -6.230503082275391, 'epoch': 0.48}

 48%|████▊     | 740/1545 [06:33<06:53,  1.95it/s]
 48%|████▊     | 741/1545 [06:34<06:56,  1.93it/s]
 48%|████▊     | 742/1545 [06:34<07:07,  1.88it/s]
 48%|████▊     | 743/1545 [06:35<06:55,  1.93it/s]
 48%|████▊     | 744/1545 [06:35<06:46,  1.97it/s]
 48%|████▊     | 745/1545 [06:36<06:56,  1.92it/s]
 48%|████▊     | 746/1545 [06:36<07:07,  1.87it/s]
 48%|████▊     | 747/1545 [06:37<06:50,  1.94it/s]
 48%|████▊     | 748/1545 [06:37<06:56,  1.91it/s]
 48%|████▊     | 749/1545 [06:38<07:07,  1.86it/s]
 49%|████▊     | 750/1545 [06:38<06:46,  1.95it/s]
                                                  
{'loss': 3.8903, 'grad_norm': 0.0, 'learning_rate': 5.145631067961165e-06, 'rewards/chosen': -22.969589233398438, 'rewards/rejected': -51.09120559692383, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 28.121618270874023, 'logps/chosen': -399.1053466796875, 'logps/rejected': -617.9463500976562, 'logits/chosen': -1.9127031564712524, 'logits/rejected': -4.489147186279297, 'epoch': 0.49}

 49%|████▊     | 750/1545 [06:38<06:46,  1.95it/s]
 49%|████▊     | 751/1545 [06:39<06:49,  1.94it/s]
 49%|████▊     | 752/1545 [06:39<07:01,  1.88it/s]
 49%|████▊     | 753/1545 [06:40<06:05,  2.17it/s]
 49%|████▉     | 754/1545 [06:40<05:17,  2.49it/s]
 49%|████▉     | 755/1545 [06:40<04:51,  2.71it/s]
 49%|████▉     | 756/1545 [06:41<04:25,  2.97it/s]
 49%|████▉     | 757/1545 [06:41<04:07,  3.19it/s]
 49%|████▉     | 758/1545 [06:41<03:55,  3.35it/s]
 49%|████▉     | 759/1545 [06:41<03:33,  3.67it/s]
 49%|████▉     | 760/1545 [06:42<03:30,  3.72it/s]
                                                  
{'loss': 0.0001, 'grad_norm': 0.0, 'learning_rate': 5.080906148867314e-06, 'rewards/chosen': -20.00014877319336, 'rewards/rejected': -70.9677505493164, 'rewards/accuracies': 1.0, 'rewards/margins': 50.96759796142578, 'logps/chosen': -339.2450256347656, 'logps/rejected': -807.701416015625, 'logits/chosen': -2.3724873065948486, 'logits/rejected': -5.234989166259766, 'epoch': 0.49}

 49%|████▉     | 760/1545 [06:42<03:30,  3.72it/s]
 49%|████▉     | 761/1545 [06:42<03:30,  3.73it/s]
 49%|████▉     | 762/1545 [06:42<03:29,  3.73it/s]
 49%|████▉     | 763/1545 [06:42<03:29,  3.73it/s]
 49%|████▉     | 764/1545 [06:43<03:31,  3.69it/s]
 50%|████▉     | 765/1545 [06:43<03:20,  3.90it/s]
 50%|████▉     | 766/1545 [06:43<03:25,  3.78it/s]
 50%|████▉     | 767/1545 [06:44<06:59,  1.86it/s]
 50%|████▉     | 768/1545 [06:45<05:56,  2.18it/s]
 50%|████▉     | 769/1545 [06:45<05:12,  2.48it/s]
 50%|████▉     | 770/1545 [06:45<04:41,  2.75it/s]
                                                  
{'loss': 1.0551, 'grad_norm': 6.261267546103788e-18, 'learning_rate': 5.016181229773464e-06, 'rewards/chosen': -21.470638275146484, 'rewards/rejected': -69.12281036376953, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 47.65216827392578, 'logps/chosen': -400.18658447265625, 'logps/rejected': -822.2199096679688, 'logits/chosen': -2.1039681434631348, 'logits/rejected': -4.775751113891602, 'epoch': 0.5}

 50%|████▉     | 770/1545 [06:45<04:41,  2.75it/s]
 50%|████▉     | 771/1545 [06:45<04:21,  2.96it/s]
 50%|████▉     | 772/1545 [06:46<04:09,  3.10it/s]
 50%|█████     | 773/1545 [06:46<03:46,  3.41it/s]
 50%|█████     | 774/1545 [06:46<03:41,  3.48it/s]
 50%|█████     | 775/1545 [06:46<03:37,  3.55it/s]
 50%|█████     | 776/1545 [06:47<03:40,  3.48it/s]
 50%|█████     | 777/1545 [06:47<03:46,  3.40it/s]
 50%|█████     | 778/1545 [06:47<03:30,  3.64it/s]
 50%|█████     | 779/1545 [06:48<03:31,  3.61it/s]
 50%|█████     | 780/1545 [06:48<03:33,  3.59it/s]
                                                  
{'loss': 0.0001, 'grad_norm': 0.0, 'learning_rate': 4.951456310679612e-06, 'rewards/chosen': -25.87270736694336, 'rewards/rejected': -69.1749267578125, 'rewards/accuracies': 1.0, 'rewards/margins': 43.302223205566406, 'logps/chosen': -421.61248779296875, 'logps/rejected': -803.4362182617188, 'logits/chosen': -2.607177972793579, 'logits/rejected': -4.537522315979004, 'epoch': 0.5}

 50%|█████     | 780/1545 [06:48<03:33,  3.59it/s]
 51%|█████     | 781/1545 [06:48<03:56,  3.23it/s]
 51%|█████     | 782/1545 [06:49<04:22,  2.90it/s]
 51%|█████     | 783/1545 [06:49<04:42,  2.70it/s]
 51%|█████     | 784/1545 [06:49<04:20,  2.92it/s]
 51%|█████     | 785/1545 [06:50<04:43,  2.68it/s]
 51%|█████     | 786/1545 [06:50<04:58,  2.54it/s]
 51%|█████     | 787/1545 [06:51<05:15,  2.40it/s]
 51%|█████     | 788/1545 [06:51<06:04,  2.08it/s]
 51%|█████     | 789/1545 [06:52<06:22,  1.97it/s]
 51%|█████     | 790/1545 [06:52<06:27,  1.95it/s]
                                                  
{'loss': 7.3323, 'grad_norm': 0.2470703125, 'learning_rate': 4.886731391585761e-06, 'rewards/chosen': -44.284454345703125, 'rewards/rejected': -73.53287506103516, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 29.2484188079834, 'logps/chosen': -594.3431396484375, 'logps/rejected': -851.0657348632812, 'logits/chosen': -2.5097155570983887, 'logits/rejected': -3.9307990074157715, 'epoch': 0.51}

 51%|█████     | 790/1545 [06:53<06:27,  1.95it/s]
 51%|█████     | 791/1545 [06:53<06:25,  1.96it/s]
 51%|█████▏    | 792/1545 [06:54<06:44,  1.86it/s]
 51%|█████▏    | 793/1545 [06:54<06:49,  1.83it/s]
 51%|█████▏    | 794/1545 [06:55<06:30,  1.92it/s]
 51%|█████▏    | 795/1545 [06:55<06:44,  1.86it/s]
 52%|█████▏    | 796/1545 [06:56<06:45,  1.85it/s]
 52%|█████▏    | 797/1545 [06:56<06:45,  1.85it/s]
 52%|█████▏    | 798/1545 [06:57<06:24,  1.94it/s]
 52%|█████▏    | 799/1545 [06:57<06:37,  1.88it/s]
 52%|█████▏    | 800/1545 [06:58<06:41,  1.86it/s]
                                                  
{'loss': 0.0002, 'grad_norm': 1.4375, 'learning_rate': 4.82200647249191e-06, 'rewards/chosen': -13.080400466918945, 'rewards/rejected': -36.216304779052734, 'rewards/accuracies': 1.0, 'rewards/margins': 23.135906219482422, 'logps/chosen': -296.13519287109375, 'logps/rejected': -475.113037109375, 'logits/chosen': -1.1140010356903076, 'logits/rejected': -2.2951102256774902, 'epoch': 0.52}

 52%|█████▏    | 800/1545 [06:58<06:41,  1.86it/s]
 52%|█████▏    | 801/1545 [06:58<06:30,  1.91it/s]
 52%|█████▏    | 802/1545 [06:59<06:42,  1.84it/s]
 52%|█████▏    | 803/1545 [07:00<06:47,  1.82it/s]
 52%|█████▏    | 804/1545 [07:00<06:51,  1.80it/s]
 52%|█████▏    | 805/1545 [07:01<06:23,  1.93it/s]
 52%|█████▏    | 806/1545 [07:01<06:40,  1.85it/s]
 52%|█████▏    | 807/1545 [07:02<06:49,  1.80it/s]
 52%|█████▏    | 808/1545 [07:02<06:32,  1.88it/s]
 52%|█████▏    | 809/1545 [07:03<05:57,  2.06it/s]
 52%|█████▏    | 810/1545 [07:03<06:18,  1.94it/s]
                                                  
{'loss': 0.0703, 'grad_norm': 9.38598532229662e-10, 'learning_rate': 4.7572815533980585e-06, 'rewards/chosen': -24.983036041259766, 'rewards/rejected': -45.387813568115234, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 20.404781341552734, 'logps/chosen': -387.3971252441406, 'logps/rejected': -552.9581909179688, 'logits/chosen': -2.1105751991271973, 'logits/rejected': -3.3740882873535156, 'epoch': 0.52}

 52%|█████▏    | 810/1545 [07:03<06:18,  1.94it/s]
 52%|█████▏    | 811/1545 [07:04<06:21,  1.92it/s]
 53%|█████▎    | 812/1545 [07:04<06:06,  2.00it/s]
 53%|█████▎    | 813/1545 [07:05<06:30,  1.88it/s]
 53%|█████▎    | 814/1545 [07:05<06:40,  1.83it/s]
 53%|█████▎    | 815/1545 [07:06<06:39,  1.83it/s]
 53%|█████▎    | 816/1545 [07:06<06:08,  1.98it/s]
 53%|█████▎    | 817/1545 [07:07<06:27,  1.88it/s]
 53%|█████▎    | 818/1545 [07:07<06:34,  1.85it/s]
 53%|█████▎    | 819/1545 [07:08<06:28,  1.87it/s]
 53%|█████▎    | 820/1545 [07:09<06:32,  1.85it/s]
                                                  
{'loss': 1.553, 'grad_norm': 0.0, 'learning_rate': 4.6925566343042074e-06, 'rewards/chosen': -23.155715942382812, 'rewards/rejected': -62.099571228027344, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 38.94385528564453, 'logps/chosen': -375.70166015625, 'logps/rejected': -735.560791015625, 'logits/chosen': -1.7601606845855713, 'logits/rejected': -3.79761004447937, 'epoch': 0.53}

 53%|█████▎    | 820/1545 [07:09<06:32,  1.85it/s]
 53%|█████▎    | 821/1545 [07:09<06:43,  1.79it/s]
 53%|█████▎    | 822/1545 [07:10<06:47,  1.77it/s]
 53%|█████▎    | 823/1545 [07:10<06:20,  1.90it/s]
 53%|█████▎    | 824/1545 [07:11<06:30,  1.85it/s]
 53%|█████▎    | 825/1545 [07:11<05:53,  2.04it/s]
 53%|█████▎    | 826/1545 [07:12<06:06,  1.96it/s]
 54%|█████▎    | 827/1545 [07:12<05:45,  2.08it/s]
 54%|█████▎    | 828/1545 [07:13<06:05,  1.96it/s]
 54%|█████▎    | 829/1545 [07:13<06:12,  1.92it/s]
 54%|█████▎    | 830/1545 [07:14<06:14,  1.91it/s]
                                                  
{'loss': 0.0695, 'grad_norm': 2.703125, 'learning_rate': 4.627831715210356e-06, 'rewards/chosen': -38.55735397338867, 'rewards/rejected': -69.79898834228516, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 31.241634368896484, 'logps/chosen': -564.8526611328125, 'logps/rejected': -828.9482421875, 'logits/chosen': -2.5433785915374756, 'logits/rejected': -3.9132227897644043, 'epoch': 0.54}

 54%|█████▎    | 830/1545 [07:14<06:14,  1.91it/s]
 54%|█████▍    | 831/1545 [07:14<06:17,  1.89it/s]
 54%|█████▍    | 832/1545 [07:15<06:26,  1.85it/s]
 54%|█████▍    | 833/1545 [07:15<06:29,  1.83it/s]
 54%|█████▍    | 834/1545 [07:16<06:03,  1.95it/s]
 54%|█████▍    | 835/1545 [07:16<06:19,  1.87it/s]
 54%|█████▍    | 836/1545 [07:17<06:26,  1.84it/s]
 54%|█████▍    | 837/1545 [07:18<06:31,  1.81it/s]
 54%|█████▍    | 838/1545 [07:18<06:25,  1.84it/s]
 54%|█████▍    | 839/1545 [07:19<06:31,  1.80it/s]
 54%|█████▍    | 840/1545 [07:19<06:33,  1.79it/s]
                                                  
{'loss': 0.0001, 'grad_norm': 3.91155481338501e-07, 'learning_rate': 4.563106796116505e-06, 'rewards/chosen': -48.7236328125, 'rewards/rejected': -81.7518539428711, 'rewards/accuracies': 1.0, 'rewards/margins': 33.028221130371094, 'logps/chosen': -654.4888916015625, 'logps/rejected': -931.0814208984375, 'logits/chosen': -2.453652858734131, 'logits/rejected': -4.23397970199585, 'epoch': 0.54}

 54%|█████▍    | 840/1545 [07:19<06:33,  1.79it/s]
 54%|█████▍    | 841/1545 [07:20<06:12,  1.89it/s]
 54%|█████▍    | 842/1545 [07:20<06:21,  1.84it/s]
 55%|█████▍    | 843/1545 [07:21<06:25,  1.82it/s]
 55%|█████▍    | 844/1545 [07:21<06:21,  1.84it/s]
 55%|█████▍    | 845/1545 [07:22<06:10,  1.89it/s]
 55%|█████▍    | 846/1545 [07:22<06:18,  1.85it/s]
 55%|█████▍    | 847/1545 [07:23<06:19,  1.84it/s]
 55%|█████▍    | 848/1545 [07:23<05:54,  1.97it/s]
 55%|█████▍    | 849/1545 [07:24<06:07,  1.89it/s]
 55%|█████▌    | 850/1545 [07:25<06:14,  1.86it/s]
                                                  
{'loss': 0.0, 'grad_norm': 2.286988957586611e-19, 'learning_rate': 4.498381877022654e-06, 'rewards/chosen': -44.718666076660156, 'rewards/rejected': -92.24812316894531, 'rewards/accuracies': 1.0, 'rewards/margins': 47.529449462890625, 'logps/chosen': -589.9989013671875, 'logps/rejected': -1017.69140625, 'logits/chosen': -3.3844847679138184, 'logits/rejected': -4.966015338897705, 'epoch': 0.55}

 55%|█████▌    | 850/1545 [07:25<06:14,  1.86it/s]
 55%|█████▌    | 851/1545 [07:25<06:21,  1.82it/s]
 55%|█████▌    | 852/1545 [07:26<05:57,  1.94it/s]
 55%|█████▌    | 853/1545 [07:26<06:09,  1.87it/s]
 55%|█████▌    | 854/1545 [07:27<06:16,  1.83it/s]
 55%|█████▌    | 855/1545 [07:27<06:05,  1.89it/s]
 55%|█████▌    | 856/1545 [07:28<06:06,  1.88it/s]
 55%|█████▌    | 857/1545 [07:28<06:19,  1.81it/s]
 56%|█████▌    | 858/1545 [07:29<06:22,  1.80it/s]
 56%|█████▌    | 859/1545 [07:29<05:49,  1.96it/s]
 56%|█████▌    | 860/1545 [07:30<06:02,  1.89it/s]
                                                  
{'loss': 0.8693, 'grad_norm': 0.0, 'learning_rate': 4.433656957928803e-06, 'rewards/chosen': -40.95757293701172, 'rewards/rejected': -79.82716369628906, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 38.86958694458008, 'logps/chosen': -559.9370727539062, 'logps/rejected': -917.9359130859375, 'logits/chosen': -3.1516611576080322, 'logits/rejected': -4.5850043296813965, 'epoch': 0.56}

 56%|█████▌    | 860/1545 [07:30<06:02,  1.89it/s]
 56%|█████▌    | 861/1545 [07:30<06:11,  1.84it/s]
 56%|█████▌    | 862/1545 [07:31<06:10,  1.85it/s]
 56%|█████▌    | 863/1545 [07:31<06:05,  1.87it/s]
 56%|█████▌    | 864/1545 [07:32<06:12,  1.83it/s]
 56%|█████▌    | 865/1545 [07:33<06:18,  1.80it/s]
 56%|█████▌    | 866/1545 [07:33<05:57,  1.90it/s]
 56%|█████▌    | 867/1545 [07:34<06:09,  1.84it/s]
 56%|█████▌    | 868/1545 [07:34<05:36,  2.01it/s]
 56%|█████▌    | 869/1545 [07:35<05:47,  1.95it/s]
 56%|█████▋    | 870/1545 [07:35<05:25,  2.07it/s]
                                                  
{'loss': 1.2715, 'grad_norm': 0.0, 'learning_rate': 4.368932038834952e-06, 'rewards/chosen': -32.4947509765625, 'rewards/rejected': -78.07811737060547, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 45.58336639404297, 'logps/chosen': -480.0074157714844, 'logps/rejected': -896.2662963867188, 'logits/chosen': -2.7316079139709473, 'logits/rejected': -4.247876167297363, 'epoch': 0.56}

 56%|█████▋    | 870/1545 [07:35<05:25,  2.07it/s]
 56%|█████▋    | 871/1545 [07:36<05:46,  1.95it/s]
 56%|█████▋    | 872/1545 [07:36<05:55,  1.89it/s]
 57%|█████▋    | 873/1545 [07:37<05:58,  1.88it/s]
 57%|█████▋    | 874/1545 [07:37<05:49,  1.92it/s]
 57%|█████▋    | 875/1545 [07:38<05:59,  1.86it/s]
 57%|█████▋    | 876/1545 [07:38<06:03,  1.84it/s]
 57%|█████▋    | 877/1545 [07:39<05:44,  1.94it/s]
 57%|█████▋    | 878/1545 [07:39<05:56,  1.87it/s]
 57%|█████▋    | 879/1545 [07:40<05:24,  2.05it/s]
 57%|█████▋    | 880/1545 [07:40<05:45,  1.93it/s]
                                                  
{'loss': 2.2344, 'grad_norm': 3.552436828613281e-05, 'learning_rate': 4.304207119741101e-06, 'rewards/chosen': -30.8316707611084, 'rewards/rejected': -63.74101638793945, 'rewards/accuracies': 0.699999988079071, 'rewards/margins': 32.909339904785156, 'logps/chosen': -454.8379821777344, 'logps/rejected': -745.82373046875, 'logits/chosen': -2.527254581451416, 'logits/rejected': -3.76324725151062, 'epoch': 0.57}

 57%|█████▋    | 880/1545 [07:40<05:45,  1.93it/s]
 57%|█████▋    | 881/1545 [07:41<05:34,  1.98it/s]
 57%|█████▋    | 882/1545 [07:41<05:45,  1.92it/s]
 57%|█████▋    | 883/1545 [07:42<05:49,  1.89it/s]
 57%|█████▋    | 884/1545 [07:42<05:48,  1.90it/s]
 57%|█████▋    | 885/1545 [07:43<05:36,  1.96it/s]
 57%|█████▋    | 886/1545 [07:44<05:55,  1.85it/s]
 57%|█████▋    | 887/1545 [07:44<05:52,  1.87it/s]
 57%|█████▋    | 888/1545 [07:45<05:39,  1.94it/s]
 58%|█████▊    | 889/1545 [07:46<08:42,  1.25it/s]
 58%|█████▊    | 890/1545 [07:47<07:52,  1.39it/s]
                                                  
{'loss': 0.0, 'grad_norm': 3.0547380447387695e-07, 'learning_rate': 4.23948220064725e-06, 'rewards/chosen': -15.884663581848145, 'rewards/rejected': -51.30836868286133, 'rewards/accuracies': 1.0, 'rewards/margins': 35.423702239990234, 'logps/chosen': -304.71734619140625, 'logps/rejected': -617.8062744140625, 'logits/chosen': -1.6831843852996826, 'logits/rejected': -3.7176902294158936, 'epoch': 0.58}

 58%|█████▊    | 890/1545 [07:47<07:52,  1.39it/s]
 58%|█████▊    | 891/1545 [07:47<07:17,  1.50it/s]
 58%|█████▊    | 892/1545 [07:48<06:43,  1.62it/s]
 58%|█████▊    | 893/1545 [07:48<06:37,  1.64it/s]
 58%|█████▊    | 894/1545 [07:49<06:27,  1.68it/s]
 58%|█████▊    | 895/1545 [07:49<05:58,  1.81it/s]
 58%|█████▊    | 896/1545 [07:50<06:07,  1.77it/s]
 58%|█████▊    | 897/1545 [07:50<06:11,  1.74it/s]
 58%|█████▊    | 898/1545 [07:51<06:01,  1.79it/s]
 58%|█████▊    | 899/1545 [07:51<05:54,  1.82it/s]
 58%|█████▊    | 900/1545 [07:52<05:58,  1.80it/s]
                                                  
{'loss': 0.0021, 'grad_norm': 2.453125, 'learning_rate': 4.1747572815533986e-06, 'rewards/chosen': -22.741947174072266, 'rewards/rejected': -60.978431701660156, 'rewards/accuracies': 1.0, 'rewards/margins': 38.236488342285156, 'logps/chosen': -365.04083251953125, 'logps/rejected': -720.0584716796875, 'logits/chosen': -2.4995644092559814, 'logits/rejected': -3.6995277404785156, 'epoch': 0.58}

 58%|█████▊    | 900/1545 [07:52<05:58,  1.80it/s]
 58%|█████▊    | 901/1545 [07:53<06:02,  1.78it/s]
 58%|█████▊    | 902/1545 [07:53<05:34,  1.92it/s]
 58%|█████▊    | 903/1545 [07:54<05:43,  1.87it/s]
 59%|█████▊    | 904/1545 [07:54<05:45,  1.86it/s]
 59%|█████▊    | 905/1545 [07:55<05:46,  1.85it/s]
 59%|█████▊    | 906/1545 [07:55<05:21,  1.99it/s]
 59%|█████▊    | 907/1545 [07:56<06:46,  1.57it/s]
 59%|█████▉    | 908/1545 [07:56<06:03,  1.75it/s]
 59%|█████▉    | 909/1545 [07:57<05:44,  1.84it/s]
 59%|█████▉    | 910/1545 [07:57<05:56,  1.78it/s]
                                                  
{'loss': 0.0, 'grad_norm': 0.001312255859375, 'learning_rate': 4.1100323624595475e-06, 'rewards/chosen': -24.250585556030273, 'rewards/rejected': -54.26154708862305, 'rewards/accuracies': 1.0, 'rewards/margins': 30.010961532592773, 'logps/chosen': -370.4151611328125, 'logps/rejected': -644.9264526367188, 'logits/chosen': -2.4921982288360596, 'logits/rejected': -3.8125457763671875, 'epoch': 0.59}

 59%|█████▉    | 910/1545 [07:58<05:56,  1.78it/s]
 59%|█████▉    | 911/1545 [07:58<05:59,  1.76it/s]
 59%|█████▉    | 912/1545 [07:58<05:20,  1.97it/s]
 59%|█████▉    | 913/1545 [07:59<05:02,  2.09it/s]
 59%|█████▉    | 914/1545 [07:59<04:43,  2.22it/s]
 59%|█████▉    | 915/1545 [08:00<05:10,  2.03it/s]
 59%|█████▉    | 916/1545 [08:00<05:22,  1.95it/s]
 59%|█████▉    | 917/1545 [08:01<04:39,  2.25it/s]
 59%|█████▉    | 918/1545 [08:01<05:09,  2.02it/s]
 59%|█████▉    | 919/1545 [08:02<05:21,  1.95it/s]
 60%|█████▉    | 920/1545 [08:02<05:27,  1.91it/s]
                                                  
{'loss': 0.0141, 'grad_norm': 3.790855407714844e-05, 'learning_rate': 4.045307443365696e-06, 'rewards/chosen': -25.498334884643555, 'rewards/rejected': -52.45038986206055, 'rewards/accuracies': 1.0, 'rewards/margins': 26.952056884765625, 'logps/chosen': -437.73626708984375, 'logps/rejected': -665.8396606445312, 'logits/chosen': -2.0662381649017334, 'logits/rejected': -3.4163360595703125, 'epoch': 0.6}

 60%|█████▉    | 920/1545 [08:02<05:27,  1.91it/s]
 60%|█████▉    | 921/1545 [08:03<05:28,  1.90it/s]
 60%|█████▉    | 922/1545 [08:04<05:37,  1.85it/s]
 60%|█████▉    | 923/1545 [08:04<05:40,  1.82it/s]
 60%|█████▉    | 924/1545 [08:04<05:16,  1.96it/s]
 60%|█████▉    | 925/1545 [08:05<05:30,  1.87it/s]
 60%|█████▉    | 926/1545 [08:06<05:42,  1.81it/s]
 60%|██████    | 927/1545 [08:06<05:38,  1.82it/s]
 60%|██████    | 928/1545 [08:07<05:29,  1.88it/s]
 60%|██████    | 929/1545 [08:07<05:41,  1.80it/s]
 60%|██████    | 930/1545 [08:08<05:35,  1.84it/s]
                                                  
{'loss': 0.0, 'grad_norm': 9.441375732421875e-05, 'learning_rate': 3.980582524271845e-06, 'rewards/chosen': -17.19916343688965, 'rewards/rejected': -57.50432586669922, 'rewards/accuracies': 1.0, 'rewards/margins': 40.305152893066406, 'logps/chosen': -328.04193115234375, 'logps/rejected': -684.6046752929688, 'logits/chosen': -2.187042236328125, 'logits/rejected': -4.470019340515137, 'epoch': 0.6}

 60%|██████    | 930/1545 [08:08<05:35,  1.84it/s]
 60%|██████    | 931/1545 [08:08<05:01,  2.03it/s]
 60%|██████    | 932/1545 [08:09<04:25,  2.31it/s]
 60%|██████    | 933/1545 [08:09<04:15,  2.39it/s]
 60%|██████    | 934/1545 [08:09<04:46,  2.13it/s]
 61%|██████    | 935/1545 [08:10<05:12,  1.95it/s]
 61%|██████    | 936/1545 [08:11<05:03,  2.01it/s]
 61%|██████    | 937/1545 [08:11<05:19,  1.91it/s]
 61%|██████    | 938/1545 [08:12<05:27,  1.85it/s]
 61%|██████    | 939/1545 [08:12<05:29,  1.84it/s]
 61%|██████    | 940/1545 [08:13<05:12,  1.94it/s]
                                                  
{'loss': 0.0009, 'grad_norm': 5.857145879417658e-10, 'learning_rate': 3.915857605177994e-06, 'rewards/chosen': -24.126405715942383, 'rewards/rejected': -60.42162322998047, 'rewards/accuracies': 1.0, 'rewards/margins': 36.29521942138672, 'logps/chosen': -361.6978454589844, 'logps/rejected': -705.1375732421875, 'logits/chosen': -2.889936923980713, 'logits/rejected': -4.388113975524902, 'epoch': 0.61}

 61%|██████    | 940/1545 [08:13<05:12,  1.94it/s]
 61%|██████    | 941/1545 [08:13<05:27,  1.84it/s]
 61%|██████    | 942/1545 [08:14<05:29,  1.83it/s]
 61%|██████    | 943/1545 [08:14<05:16,  1.90it/s]
 61%|██████    | 944/1545 [08:15<05:22,  1.86it/s]
 61%|██████    | 945/1545 [08:15<05:26,  1.84it/s]
 61%|██████    | 946/1545 [08:16<05:34,  1.79it/s]
 61%|██████▏   | 947/1545 [08:16<05:08,  1.94it/s]
 61%|██████▏   | 948/1545 [08:17<05:22,  1.85it/s]
 61%|██████▏   | 949/1545 [08:18<05:25,  1.83it/s]
 61%|██████▏   | 950/1545 [08:18<05:18,  1.87it/s]
                                                  
{'loss': 0.1108, 'grad_norm': 3.202843331805323e-21, 'learning_rate': 3.851132686084142e-06, 'rewards/chosen': -30.862689971923828, 'rewards/rejected': -68.66695404052734, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 37.804264068603516, 'logps/chosen': -447.46221923828125, 'logps/rejected': -795.8951416015625, 'logits/chosen': -2.5695688724517822, 'logits/rejected': -4.074126243591309, 'epoch': 0.61}

 61%|██████▏   | 950/1545 [08:18<05:18,  1.87it/s]
 62%|██████▏   | 951/1545 [08:19<05:24,  1.83it/s]
 62%|██████▏   | 952/1545 [08:19<05:27,  1.81it/s]
 62%|██████▏   | 953/1545 [08:20<05:30,  1.79it/s]
 62%|██████▏   | 954/1545 [08:20<05:07,  1.92it/s]
 62%|██████▏   | 955/1545 [08:21<05:16,  1.87it/s]
 62%|██████▏   | 956/1545 [08:21<05:17,  1.85it/s]
 62%|██████▏   | 957/1545 [08:22<05:15,  1.86it/s]
 62%|██████▏   | 958/1545 [08:22<05:07,  1.91it/s]
 62%|██████▏   | 959/1545 [08:23<05:18,  1.84it/s]
 62%|██████▏   | 960/1545 [08:24<05:23,  1.81it/s]
                                                  
{'loss': 0.0938, 'grad_norm': 1.7848833522293717e-11, 'learning_rate': 3.7864077669902915e-06, 'rewards/chosen': -31.656015396118164, 'rewards/rejected': -76.21475982666016, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 44.558738708496094, 'logps/chosen': -445.54949951171875, 'logps/rejected': -868.03564453125, 'logits/chosen': -2.807201862335205, 'logits/rejected': -4.515857696533203, 'epoch': 0.62}

 62%|██████▏   | 960/1545 [08:24<05:23,  1.81it/s]
 62%|██████▏   | 961/1545 [08:24<05:05,  1.91it/s]
 62%|██████▏   | 962/1545 [08:25<05:18,  1.83it/s]
 62%|██████▏   | 963/1545 [08:25<05:20,  1.81it/s]
 62%|██████▏   | 964/1545 [08:26<05:18,  1.83it/s]
 62%|██████▏   | 965/1545 [08:26<05:09,  1.88it/s]
 63%|██████▎   | 966/1545 [08:27<05:17,  1.82it/s]
 63%|██████▎   | 967/1545 [08:27<05:22,  1.79it/s]
 63%|██████▎   | 968/1545 [08:28<05:00,  1.92it/s]
 63%|██████▎   | 969/1545 [08:28<05:17,  1.81it/s]
 63%|██████▎   | 970/1545 [08:29<05:20,  1.80it/s]
                                                  
{'loss': 0.0, 'grad_norm': 1.8596649169921875e-05, 'learning_rate': 3.721682847896441e-06, 'rewards/chosen': -31.059711456298828, 'rewards/rejected': -74.49284362792969, 'rewards/accuracies': 1.0, 'rewards/margins': 43.43313217163086, 'logps/chosen': -468.68341064453125, 'logps/rejected': -852.8173828125, 'logits/chosen': -2.2017874717712402, 'logits/rejected': -4.380518913269043, 'epoch': 0.63}

 63%|██████▎   | 970/1545 [08:29<05:20,  1.80it/s]
 63%|██████▎   | 971/1545 [08:29<04:51,  1.97it/s]
 63%|██████▎   | 972/1545 [08:30<04:11,  2.28it/s]
 63%|██████▎   | 973/1545 [08:30<04:36,  2.07it/s]
 63%|██████▎   | 974/1545 [08:31<04:48,  1.98it/s]
 63%|██████▎   | 975/1545 [08:31<04:57,  1.91it/s]
 63%|██████▎   | 976/1545 [08:32<04:36,  2.06it/s]
 63%|██████▎   | 977/1545 [08:32<04:51,  1.95it/s]
 63%|██████▎   | 978/1545 [08:33<04:56,  1.91it/s]
 63%|██████▎   | 979/1545 [08:33<04:58,  1.90it/s]
 63%|██████▎   | 980/1545 [08:34<04:52,  1.93it/s]
                                                  
{'loss': 0.0, 'grad_norm': 2.656295322589486e-17, 'learning_rate': 3.6569579288025893e-06, 'rewards/chosen': -23.480316162109375, 'rewards/rejected': -78.44859313964844, 'rewards/accuracies': 1.0, 'rewards/margins': 54.9682731628418, 'logps/chosen': -396.80621337890625, 'logps/rejected': -915.10791015625, 'logits/chosen': -2.1880345344543457, 'logits/rejected': -4.083585739135742, 'epoch': 0.63}

 63%|██████▎   | 980/1545 [08:34<04:52,  1.93it/s]
 63%|██████▎   | 981/1545 [08:35<05:05,  1.84it/s]
 64%|██████▎   | 982/1545 [08:35<05:08,  1.83it/s]
 64%|██████▎   | 983/1545 [08:36<04:48,  1.94it/s]
 64%|██████▎   | 984/1545 [08:36<05:01,  1.86it/s]
 64%|██████▍   | 985/1545 [08:37<05:05,  1.84it/s]
 64%|██████▍   | 986/1545 [08:37<05:01,  1.85it/s]
 64%|██████▍   | 987/1545 [08:38<04:48,  1.93it/s]
 64%|██████▍   | 988/1545 [08:38<04:56,  1.88it/s]
 64%|██████▍   | 989/1545 [08:39<05:02,  1.84it/s]
 64%|██████▍   | 990/1545 [08:39<04:52,  1.90it/s]
                                                  
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 3.592233009708738e-06, 'rewards/chosen': -31.23908042907715, 'rewards/rejected': -84.39486694335938, 'rewards/accuracies': 1.0, 'rewards/margins': 53.155784606933594, 'logps/chosen': -469.037109375, 'logps/rejected': -947.4615478515625, 'logits/chosen': -2.7216758728027344, 'logits/rejected': -4.766693115234375, 'epoch': 0.64}

 64%|██████▍   | 990/1545 [08:39<04:52,  1.90it/s]
 64%|██████▍   | 991/1545 [08:40<05:05,  1.81it/s]
 64%|██████▍   | 992/1545 [08:41<05:06,  1.80it/s]
 64%|██████▍   | 993/1545 [08:41<04:36,  2.00it/s]
 64%|██████▍   | 994/1545 [08:41<04:24,  2.08it/s]
 64%|██████▍   | 995/1545 [08:42<04:40,  1.96it/s]
 64%|██████▍   | 996/1545 [08:42<04:50,  1.89it/s]
 65%|██████▍   | 997/1545 [08:43<04:52,  1.88it/s]
 65%|██████▍   | 998/1545 [08:43<04:31,  2.02it/s]
 65%|██████▍   | 999/1545 [08:44<04:45,  1.91it/s]
 65%|██████▍   | 1000/1545 [08:45<04:48,  1.89it/s]
                                                   
{'loss': 0.0, 'grad_norm': 2.8189256484623115e-18, 'learning_rate': 3.5275080906148866e-06, 'rewards/chosen': -29.645822525024414, 'rewards/rejected': -76.21923065185547, 'rewards/accuracies': 1.0, 'rewards/margins': 46.57341384887695, 'logps/chosen': -474.364990234375, 'logps/rejected': -886.2589721679688, 'logits/chosen': -2.4084129333496094, 'logits/rejected': -3.961566209793091, 'epoch': 0.65}

 65%|██████▍   | 1000/1545 [08:45<04:48,  1.89it/s]
 65%|██████▍   | 1001/1545 [08:45<04:48,  1.88it/s]
 65%|██████▍   | 1002/1545 [08:46<04:52,  1.86it/s]
 65%|██████▍   | 1003/1545 [08:47<07:29,  1.20it/s]
 65%|██████▍   | 1004/1545 [08:48<06:42,  1.34it/s]
 65%|██████▌   | 1005/1545 [08:48<05:51,  1.54it/s]
 65%|██████▌   | 1006/1545 [08:49<05:40,  1.58it/s]
 65%|██████▌   | 1007/1545 [08:49<05:29,  1.63it/s]
 65%|██████▌   | 1008/1545 [08:50<05:10,  1.73it/s]
 65%|██████▌   | 1009/1545 [08:50<05:14,  1.70it/s]
 65%|██████▌   | 1010/1545 [08:51<05:14,  1.70it/s]
                                                   
{'loss': 0.0, 'grad_norm': 6.352747104407253e-22, 'learning_rate': 3.462783171521036e-06, 'rewards/chosen': -32.23039245605469, 'rewards/rejected': -103.32981872558594, 'rewards/accuracies': 1.0, 'rewards/margins': 71.09942626953125, 'logps/chosen': -480.616455078125, 'logps/rejected': -1155.624267578125, 'logits/chosen': -2.7920470237731934, 'logits/rejected': -4.624792575836182, 'epoch': 0.65}

 65%|██████▌   | 1010/1545 [08:51<05:14,  1.70it/s]
 65%|██████▌   | 1011/1545 [08:52<05:09,  1.73it/s]
 66%|██████▌   | 1012/1545 [08:52<04:51,  1.83it/s]
 66%|██████▌   | 1013/1545 [08:53<04:57,  1.79it/s]
 66%|██████▌   | 1014/1545 [08:53<04:57,  1.79it/s]
 66%|██████▌   | 1015/1545 [08:54<04:42,  1.88it/s]
 66%|██████▌   | 1016/1545 [08:54<04:18,  2.05it/s]
 66%|██████▌   | 1017/1545 [08:54<04:00,  2.20it/s]
 66%|██████▌   | 1018/1545 [08:55<04:18,  2.04it/s]
 66%|██████▌   | 1019/1545 [08:56<04:24,  1.99it/s]
 66%|██████▌   | 1020/1545 [08:56<04:24,  1.99it/s]
                                                   
{'loss': 0.0, 'grad_norm': 1.895427703857422e-05, 'learning_rate': 3.398058252427185e-06, 'rewards/chosen': -47.92738342285156, 'rewards/rejected': -90.48072052001953, 'rewards/accuracies': 1.0, 'rewards/margins': 42.553340911865234, 'logps/chosen': -608.0487060546875, 'logps/rejected': -1020.53955078125, 'logits/chosen': -3.5880520343780518, 'logits/rejected': -4.89176607131958, 'epoch': 0.66}

 66%|██████▌   | 1020/1545 [08:56<04:24,  1.99it/s]
 66%|██████▌   | 1021/1545 [08:57<04:38,  1.88it/s]
 66%|██████▌   | 1022/1545 [08:57<04:45,  1.83it/s]
 66%|██████▌   | 1023/1545 [08:58<04:24,  1.97it/s]
 66%|██████▋   | 1024/1545 [08:58<04:35,  1.89it/s]
 66%|██████▋   | 1025/1545 [08:59<04:41,  1.85it/s]
 66%|██████▋   | 1026/1545 [08:59<04:38,  1.86it/s]
 66%|██████▋   | 1027/1545 [09:00<04:26,  1.94it/s]
 67%|██████▋   | 1028/1545 [09:00<04:37,  1.86it/s]
 67%|██████▋   | 1029/1545 [09:01<04:41,  1.84it/s]
 67%|██████▋   | 1030/1545 [09:01<04:12,  2.04it/s]
                                                   
{'loss': 0.0067, 'grad_norm': 0.000110626220703125, 'learning_rate': 3.3333333333333333e-06, 'rewards/chosen': -31.464359283447266, 'rewards/rejected': -79.84764862060547, 'rewards/accuracies': 1.0, 'rewards/margins': 48.38329315185547, 'logps/chosen': -460.6871032714844, 'logps/rejected': -919.3955078125, 'logits/chosen': -2.863114833831787, 'logits/rejected': -4.340217113494873, 'epoch': 0.67}

 67%|██████▋   | 1030/1545 [09:01<04:12,  2.04it/s]
 67%|██████▋   | 1031/1545 [09:02<04:15,  2.01it/s]
 67%|██████▋   | 1032/1545 [09:02<04:26,  1.92it/s]
 67%|██████▋   | 1033/1545 [09:03<04:29,  1.90it/s]
 67%|██████▋   | 1034/1545 [09:03<04:16,  1.99it/s]
 67%|██████▋   | 1035/1545 [09:04<04:28,  1.90it/s]
 67%|██████▋   | 1036/1545 [09:04<04:33,  1.86it/s]
 67%|██████▋   | 1037/1545 [09:05<04:35,  1.84it/s]
 67%|██████▋   | 1038/1545 [09:05<04:20,  1.95it/s]
 67%|██████▋   | 1039/1545 [09:06<04:29,  1.88it/s]
 67%|██████▋   | 1040/1545 [09:07<04:34,  1.84it/s]
                                                   
{'loss': 0.0878, 'grad_norm': 2.0161650127192843e-13, 'learning_rate': 3.2686084142394826e-06, 'rewards/chosen': -22.702207565307617, 'rewards/rejected': -70.41169738769531, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 47.7094841003418, 'logps/chosen': -373.5330810546875, 'logps/rejected': -809.3206787109375, 'logits/chosen': -2.131121873855591, 'logits/rejected': -4.659956932067871, 'epoch': 0.67}

 67%|██████▋   | 1040/1545 [09:07<04:34,  1.84it/s]
 67%|██████▋   | 1041/1545 [09:07<04:37,  1.81it/s]
 67%|██████▋   | 1042/1545 [09:08<05:11,  1.61it/s]
 68%|██████▊   | 1043/1545 [09:09<05:29,  1.52it/s]
 68%|██████▊   | 1044/1545 [09:09<05:04,  1.65it/s]
 68%|██████▊   | 1045/1545 [09:10<04:30,  1.85it/s]
 68%|██████▊   | 1046/1545 [09:10<04:49,  1.73it/s]
 68%|██████▊   | 1047/1545 [09:11<04:48,  1.73it/s]
 68%|██████▊   | 1048/1545 [09:11<04:22,  1.89it/s]
 68%|██████▊   | 1049/1545 [09:12<04:33,  1.81it/s]
 68%|██████▊   | 1050/1545 [09:12<04:05,  2.01it/s]
                                                   
{'loss': 0.0693, 'grad_norm': 1.5802470443304628e-11, 'learning_rate': 3.2038834951456315e-06, 'rewards/chosen': -28.963123321533203, 'rewards/rejected': -69.37215423583984, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 40.409034729003906, 'logps/chosen': -429.06085205078125, 'logps/rejected': -803.1228637695312, 'logits/chosen': -2.533982038497925, 'logits/rejected': -4.221534252166748, 'epoch': 0.68}

 68%|██████▊   | 1050/1545 [09:12<04:05,  2.01it/s]
 68%|██████▊   | 1051/1545 [09:13<04:13,  1.95it/s]
 68%|██████▊   | 1052/1545 [09:13<04:01,  2.04it/s]
 68%|██████▊   | 1053/1545 [09:14<04:16,  1.92it/s]
 68%|██████▊   | 1054/1545 [09:14<04:23,  1.86it/s]
 68%|██████▊   | 1055/1545 [09:15<04:14,  1.93it/s]
 68%|██████▊   | 1056/1545 [09:15<04:21,  1.87it/s]
 68%|██████▊   | 1057/1545 [09:16<04:26,  1.83it/s]
 68%|██████▊   | 1058/1545 [09:17<04:28,  1.81it/s]
 69%|██████▊   | 1059/1545 [09:17<04:07,  1.96it/s]
 69%|██████▊   | 1060/1545 [09:18<04:14,  1.90it/s]
                                                   
{'loss': 0.0598, 'grad_norm': 0.0, 'learning_rate': 3.13915857605178e-06, 'rewards/chosen': -34.7342643737793, 'rewards/rejected': -71.29218292236328, 'rewards/accuracies': 1.0, 'rewards/margins': 36.55791091918945, 'logps/chosen': -516.7052612304688, 'logps/rejected': -824.2127685546875, 'logits/chosen': -3.0466580390930176, 'logits/rejected': -4.670151710510254, 'epoch': 0.69}

 69%|██████▊   | 1060/1545 [09:18<04:14,  1.90it/s]
 69%|██████▊   | 1061/1545 [09:18<04:19,  1.86it/s]
 69%|██████▊   | 1062/1545 [09:19<04:21,  1.85it/s]
 69%|██████▉   | 1063/1545 [09:19<04:43,  1.70it/s]
 69%|██████▉   | 1064/1545 [09:20<05:05,  1.57it/s]
 69%|██████▉   | 1065/1545 [09:21<04:41,  1.70it/s]
 69%|██████▉   | 1066/1545 [09:21<04:41,  1.70it/s]
 69%|██████▉   | 1067/1545 [09:22<04:38,  1.72it/s]
 69%|██████▉   | 1068/1545 [09:22<04:35,  1.73it/s]
 69%|██████▉   | 1069/1545 [09:23<04:17,  1.85it/s]
 69%|██████▉   | 1070/1545 [09:23<04:22,  1.81it/s]
                                                   
{'loss': 0.0, 'grad_norm': 6.733911930671688e-20, 'learning_rate': 3.0744336569579293e-06, 'rewards/chosen': -25.6965274810791, 'rewards/rejected': -69.01310729980469, 'rewards/accuracies': 1.0, 'rewards/margins': 43.31658172607422, 'logps/chosen': -386.03173828125, 'logps/rejected': -804.2999877929688, 'logits/chosen': -2.817981004714966, 'logits/rejected': -4.601845741271973, 'epoch': 0.69}

 69%|██████▉   | 1070/1545 [09:23<04:22,  1.81it/s]
 69%|██████▉   | 1071/1545 [09:24<04:26,  1.78it/s]
 69%|██████▉   | 1072/1545 [09:24<04:21,  1.81it/s]
 69%|██████▉   | 1073/1545 [09:25<04:17,  1.83it/s]
 70%|██████▉   | 1074/1545 [09:26<04:21,  1.80it/s]
 70%|██████▉   | 1075/1545 [09:26<04:22,  1.79it/s]
 70%|██████▉   | 1076/1545 [09:27<04:06,  1.90it/s]
 70%|██████▉   | 1077/1545 [09:27<04:15,  1.83it/s]
 70%|██████▉   | 1078/1545 [09:28<04:16,  1.82it/s]
 70%|██████▉   | 1079/1545 [09:28<04:14,  1.83it/s]
 70%|██████▉   | 1080/1545 [09:29<04:08,  1.87it/s]
                                                   
{'loss': 0.0, 'grad_norm': 4.3298697960381105e-15, 'learning_rate': 3.0097087378640778e-06, 'rewards/chosen': -30.43539047241211, 'rewards/rejected': -81.25576782226562, 'rewards/accuracies': 1.0, 'rewards/margins': 50.82037353515625, 'logps/chosen': -435.7405700683594, 'logps/rejected': -922.3958740234375, 'logits/chosen': -3.113784074783325, 'logits/rejected': -4.786489963531494, 'epoch': 0.7}

 70%|██████▉   | 1080/1545 [09:29<04:08,  1.87it/s]
 70%|██████▉   | 1081/1545 [09:29<04:16,  1.81it/s]
 70%|███████   | 1082/1545 [09:30<04:14,  1.82it/s]
 70%|███████   | 1083/1545 [09:30<03:58,  1.93it/s]
 70%|███████   | 1084/1545 [09:31<04:07,  1.86it/s]
 70%|███████   | 1085/1545 [09:32<04:12,  1.82it/s]
 70%|███████   | 1086/1545 [09:32<04:11,  1.82it/s]
 70%|███████   | 1087/1545 [09:33<04:09,  1.84it/s]
 70%|███████   | 1088/1545 [09:33<04:12,  1.81it/s]
 70%|███████   | 1089/1545 [09:34<04:12,  1.80it/s]
 71%|███████   | 1090/1545 [09:34<03:54,  1.94it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2.9449838187702267e-06, 'rewards/chosen': -32.56324005126953, 'rewards/rejected': -83.65999603271484, 'rewards/accuracies': 1.0, 'rewards/margins': 51.09675979614258, 'logps/chosen': -480.71697998046875, 'logps/rejected': -947.0802612304688, 'logits/chosen': -2.62692928314209, 'logits/rejected': -4.6637372970581055, 'epoch': 0.71}

 71%|███████   | 1090/1545 [09:34<03:54,  1.94it/s]
 71%|███████   | 1091/1545 [09:35<04:03,  1.87it/s]
 71%|███████   | 1092/1545 [09:35<04:06,  1.84it/s]
 71%|███████   | 1093/1545 [09:36<04:03,  1.86it/s]
 71%|███████   | 1094/1545 [09:36<03:48,  1.97it/s]
 71%|███████   | 1095/1545 [09:37<03:56,  1.90it/s]
 71%|███████   | 1096/1545 [09:37<04:01,  1.86it/s]
 71%|███████   | 1097/1545 [09:38<03:55,  1.91it/s]
 71%|███████   | 1098/1545 [09:38<03:56,  1.89it/s]
 71%|███████   | 1099/1545 [09:39<03:35,  2.07it/s]
 71%|███████   | 1100/1545 [09:39<03:47,  1.96it/s]
                                                   
{'loss': 0.0, 'grad_norm': 1.8925892415213273e-21, 'learning_rate': 2.880258899676376e-06, 'rewards/chosen': -30.453670501708984, 'rewards/rejected': -84.54370880126953, 'rewards/accuracies': 1.0, 'rewards/margins': 54.09003448486328, 'logps/chosen': -462.65472412109375, 'logps/rejected': -947.7019653320312, 'logits/chosen': -2.7360405921936035, 'logits/rejected': -4.802727699279785, 'epoch': 0.71}

 71%|███████   | 1100/1545 [09:39<03:47,  1.96it/s]
 71%|███████▏  | 1101/1545 [09:40<03:39,  2.02it/s]
 71%|███████▏  | 1102/1545 [09:40<03:50,  1.92it/s]
 71%|███████▏  | 1103/1545 [09:41<04:02,  1.82it/s]
 71%|███████▏  | 1104/1545 [09:41<03:38,  2.02it/s]
 72%|███████▏  | 1105/1545 [09:42<03:30,  2.09it/s]
 72%|███████▏  | 1106/1545 [09:42<03:46,  1.94it/s]
 72%|███████▏  | 1107/1545 [09:43<03:51,  1.89it/s]
 72%|███████▏  | 1108/1545 [09:44<03:48,  1.91it/s]
 72%|███████▏  | 1109/1545 [09:44<03:18,  2.20it/s]
 72%|███████▏  | 1110/1545 [09:44<03:36,  2.01it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2.8155339805825245e-06, 'rewards/chosen': -17.798450469970703, 'rewards/rejected': -69.56123352050781, 'rewards/accuracies': 1.0, 'rewards/margins': 51.76277542114258, 'logps/chosen': -317.5895690917969, 'logps/rejected': -803.85595703125, 'logits/chosen': -2.070510149002075, 'logits/rejected': -4.086310863494873, 'epoch': 0.72}

 72%|███████▏  | 1110/1545 [09:44<03:36,  2.01it/s]
 72%|███████▏  | 1111/1545 [09:45<03:47,  1.91it/s]
 72%|███████▏  | 1112/1545 [09:46<03:50,  1.88it/s]
 72%|███████▏  | 1113/1545 [09:46<03:42,  1.94it/s]
 72%|███████▏  | 1114/1545 [09:47<03:50,  1.87it/s]
 72%|███████▏  | 1115/1545 [09:47<03:51,  1.85it/s]
 72%|███████▏  | 1116/1545 [09:48<05:18,  1.35it/s]
 72%|███████▏  | 1117/1545 [09:49<05:06,  1.40it/s]
 72%|███████▏  | 1118/1545 [09:50<04:49,  1.48it/s]
 72%|███████▏  | 1119/1545 [09:50<04:30,  1.58it/s]
 72%|███████▏  | 1120/1545 [09:51<04:27,  1.59it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2.7508090614886734e-06, 'rewards/chosen': -21.955896377563477, 'rewards/rejected': -73.12271881103516, 'rewards/accuracies': 1.0, 'rewards/margins': 51.166812896728516, 'logps/chosen': -381.0746154785156, 'logps/rejected': -851.5888671875, 'logits/chosen': -2.0508859157562256, 'logits/rejected': -4.157201290130615, 'epoch': 0.72}

 72%|███████▏  | 1120/1545 [09:51<04:27,  1.59it/s]
 73%|███████▎  | 1121/1545 [09:51<04:25,  1.59it/s]
 73%|███████▎  | 1122/1545 [09:52<04:16,  1.65it/s]
 73%|███████▎  | 1123/1545 [09:52<04:07,  1.70it/s]
 73%|███████▎  | 1124/1545 [09:53<04:10,  1.68it/s]
 73%|███████▎  | 1125/1545 [09:54<04:03,  1.73it/s]
 73%|███████▎  | 1126/1545 [09:54<03:45,  1.86it/s]
 73%|███████▎  | 1127/1545 [09:55<03:53,  1.79it/s]
 73%|███████▎  | 1128/1545 [09:55<03:53,  1.79it/s]
 73%|███████▎  | 1129/1545 [09:56<03:49,  1.81it/s]
 73%|███████▎  | 1130/1545 [09:56<03:49,  1.81it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2.686084142394822e-06, 'rewards/chosen': -28.051372528076172, 'rewards/rejected': -90.2176284790039, 'rewards/accuracies': 1.0, 'rewards/margins': 62.1662483215332, 'logps/chosen': -435.447509765625, 'logps/rejected': -1026.18798828125, 'logits/chosen': -2.331815242767334, 'logits/rejected': -4.296026706695557, 'epoch': 0.73}

 73%|███████▎  | 1130/1545 [09:56<03:49,  1.81it/s]
 73%|███████▎  | 1131/1545 [09:57<03:51,  1.78it/s]
 73%|███████▎  | 1132/1545 [09:57<03:54,  1.76it/s]
 73%|███████▎  | 1133/1545 [09:58<03:37,  1.89it/s]
 73%|███████▎  | 1134/1545 [09:59<03:44,  1.83it/s]
 73%|███████▎  | 1135/1545 [09:59<03:46,  1.81it/s]
 74%|███████▎  | 1136/1545 [10:00<03:44,  1.82it/s]
 74%|███████▎  | 1137/1545 [10:00<03:39,  1.86it/s]
 74%|███████▎  | 1138/1545 [10:01<03:42,  1.83it/s]
 74%|███████▎  | 1139/1545 [10:01<03:44,  1.81it/s]
 74%|███████▍  | 1140/1545 [10:02<03:30,  1.93it/s]
                                                   
{'loss': 0.0, 'grad_norm': 8.348877145181177e-14, 'learning_rate': 2.621359223300971e-06, 'rewards/chosen': -28.582677841186523, 'rewards/rejected': -89.46198272705078, 'rewards/accuracies': 1.0, 'rewards/margins': 60.879295349121094, 'logps/chosen': -428.8086853027344, 'logps/rejected': -1000.7501831054688, 'logits/chosen': -2.752683162689209, 'logits/rejected': -4.274931907653809, 'epoch': 0.74}

 74%|███████▍  | 1140/1545 [10:02<03:30,  1.93it/s]
 74%|███████▍  | 1141/1545 [10:02<03:39,  1.84it/s]
 74%|███████▍  | 1142/1545 [10:03<03:40,  1.83it/s]
 74%|███████▍  | 1143/1545 [10:03<03:38,  1.84it/s]
 74%|███████▍  | 1144/1545 [10:04<03:29,  1.92it/s]
 74%|███████▍  | 1145/1545 [10:04<03:36,  1.85it/s]
 74%|███████▍  | 1146/1545 [10:05<03:34,  1.86it/s]
 74%|███████▍  | 1147/1545 [10:05<03:24,  1.94it/s]
 74%|███████▍  | 1148/1545 [10:06<03:35,  1.85it/s]
 74%|███████▍  | 1149/1545 [10:07<03:37,  1.82it/s]
 74%|███████▍  | 1150/1545 [10:07<03:37,  1.81it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 2.55663430420712e-06, 'rewards/chosen': -30.311452865600586, 'rewards/rejected': -88.15357971191406, 'rewards/accuracies': 1.0, 'rewards/margins': 57.84212112426758, 'logps/chosen': -434.56097412109375, 'logps/rejected': -994.1755981445312, 'logits/chosen': -2.7379162311553955, 'logits/rejected': -4.308984279632568, 'epoch': 0.74}

 74%|███████▍  | 1150/1545 [10:07<03:37,  1.81it/s]
 74%|███████▍  | 1151/1545 [10:08<03:25,  1.91it/s]
 75%|███████▍  | 1152/1545 [10:08<03:31,  1.86it/s]
 75%|███████▍  | 1153/1545 [10:09<03:33,  1.84it/s]
 75%|███████▍  | 1154/1545 [10:09<03:26,  1.89it/s]
 75%|███████▍  | 1155/1545 [10:10<03:32,  1.84it/s]
 75%|███████▍  | 1156/1545 [10:10<03:35,  1.81it/s]
 75%|███████▍  | 1157/1545 [10:11<03:34,  1.81it/s]
 75%|███████▍  | 1158/1545 [10:11<03:16,  1.97it/s]
 75%|███████▌  | 1159/1545 [10:12<03:27,  1.86it/s]
 75%|███████▌  | 1160/1545 [10:13<03:27,  1.86it/s]
                                                   
{'loss': 5.2559, 'grad_norm': 1.4543533325195312e-05, 'learning_rate': 2.491909385113269e-06, 'rewards/chosen': -36.78795623779297, 'rewards/rejected': -50.73297119140625, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 13.945019721984863, 'logps/chosen': -489.45684814453125, 'logps/rejected': -604.6040649414062, 'logits/chosen': -2.9106650352478027, 'logits/rejected': -3.734529972076416, 'epoch': 0.75}

 75%|███████▌  | 1160/1545 [10:13<03:27,  1.86it/s]
 75%|███████▌  | 1161/1545 [10:13<03:25,  1.86it/s]
 75%|███████▌  | 1162/1545 [10:14<03:53,  1.64it/s]
 75%|███████▌  | 1163/1545 [10:14<03:47,  1.68it/s]
 75%|███████▌  | 1164/1545 [10:15<03:38,  1.75it/s]
 75%|███████▌  | 1165/1545 [10:15<03:30,  1.81it/s]
 75%|███████▌  | 1166/1545 [10:16<03:31,  1.79it/s]
 76%|███████▌  | 1167/1545 [10:17<03:31,  1.79it/s]
 76%|███████▌  | 1168/1545 [10:17<03:16,  1.92it/s]
 76%|███████▌  | 1169/1545 [10:18<03:23,  1.85it/s]
 76%|███████▌  | 1170/1545 [10:18<03:23,  1.84it/s]
                                                   
{'loss': 2.6137, 'grad_norm': 4.929390229335695e-14, 'learning_rate': 2.427184466019418e-06, 'rewards/chosen': -25.188079833984375, 'rewards/rejected': -58.6072998046875, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 33.419219970703125, 'logps/chosen': -412.2906188964844, 'logps/rejected': -685.932861328125, 'logits/chosen': -1.7462126016616821, 'logits/rejected': -3.5213539600372314, 'epoch': 0.76}

 76%|███████▌  | 1170/1545 [10:18<03:23,  1.84it/s]
 76%|███████▌  | 1171/1545 [10:19<03:22,  1.85it/s]
 76%|███████▌  | 1172/1545 [10:19<03:15,  1.91it/s]
 76%|███████▌  | 1173/1545 [10:20<03:21,  1.84it/s]
 76%|███████▌  | 1174/1545 [10:20<03:18,  1.87it/s]
 76%|███████▌  | 1175/1545 [10:21<03:11,  1.93it/s]
 76%|███████▌  | 1176/1545 [10:21<03:18,  1.86it/s]
 76%|███████▌  | 1177/1545 [10:22<03:21,  1.83it/s]
 76%|███████▌  | 1178/1545 [10:22<03:22,  1.81it/s]
 76%|███████▋  | 1179/1545 [10:23<03:08,  1.95it/s]
 76%|███████▋  | 1180/1545 [10:23<03:17,  1.85it/s]
                                                   
{'loss': 0.6655, 'grad_norm': 0.0002689361572265625, 'learning_rate': 2.3624595469255667e-06, 'rewards/chosen': -22.50977325439453, 'rewards/rejected': -50.339866638183594, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 27.830097198486328, 'logps/chosen': -364.30804443359375, 'logps/rejected': -620.8726806640625, 'logits/chosen': -2.1491000652313232, 'logits/rejected': -3.5545706748962402, 'epoch': 0.76}

 76%|███████▋  | 1180/1545 [10:23<03:17,  1.85it/s]
 76%|███████▋  | 1181/1545 [10:24<03:19,  1.82it/s]
 77%|███████▋  | 1182/1545 [10:25<03:12,  1.88it/s]
 77%|███████▋  | 1183/1545 [10:25<03:15,  1.85it/s]
 77%|███████▋  | 1184/1545 [10:25<02:57,  2.04it/s]
 77%|███████▋  | 1185/1545 [10:26<03:04,  1.95it/s]
 77%|███████▋  | 1186/1545 [10:26<02:57,  2.02it/s]
 77%|███████▋  | 1187/1545 [10:27<03:06,  1.92it/s]
 77%|███████▋  | 1188/1545 [10:28<03:13,  1.85it/s]
 77%|███████▋  | 1189/1545 [10:28<02:54,  2.04it/s]
 77%|███████▋  | 1190/1545 [10:28<02:49,  2.09it/s]
                                                   
{'loss': 0.6483, 'grad_norm': 6.4375, 'learning_rate': 2.297734627831715e-06, 'rewards/chosen': -29.0391788482666, 'rewards/rejected': -50.39924240112305, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 21.360071182250977, 'logps/chosen': -452.758056640625, 'logps/rejected': -620.355712890625, 'logits/chosen': -2.2422232627868652, 'logits/rejected': -2.9050989151000977, 'epoch': 0.77}

 77%|███████▋  | 1190/1545 [10:28<02:49,  2.09it/s]
 77%|███████▋  | 1191/1545 [10:29<03:02,  1.94it/s]
 77%|███████▋  | 1192/1545 [10:29<02:47,  2.11it/s]
 77%|███████▋  | 1193/1545 [10:30<02:56,  1.99it/s]
 77%|███████▋  | 1194/1545 [10:30<02:49,  2.07it/s]
 77%|███████▋  | 1195/1545 [10:31<02:58,  1.96it/s]
 77%|███████▋  | 1196/1545 [10:32<03:02,  1.91it/s]
 77%|███████▋  | 1197/1545 [10:32<03:03,  1.89it/s]
 78%|███████▊  | 1198/1545 [10:33<02:56,  1.96it/s]
 78%|███████▊  | 1199/1545 [10:33<03:03,  1.89it/s]
 78%|███████▊  | 1200/1545 [10:34<03:06,  1.85it/s]
                                                   
{'loss': 0.0049, 'grad_norm': 0.119140625, 'learning_rate': 2.2330097087378645e-06, 'rewards/chosen': -24.393327713012695, 'rewards/rejected': -47.943050384521484, 'rewards/accuracies': 1.0, 'rewards/margins': 23.549720764160156, 'logps/chosen': -420.3701171875, 'logps/rejected': -596.0790405273438, 'logits/chosen': -2.2110846042633057, 'logits/rejected': -3.3869075775146484, 'epoch': 0.78}

 78%|███████▊  | 1200/1545 [10:34<03:06,  1.85it/s]
 78%|███████▊  | 1201/1545 [10:34<02:58,  1.93it/s]
 78%|███████▊  | 1202/1545 [10:35<03:03,  1.87it/s]
 78%|███████▊  | 1203/1545 [10:35<03:08,  1.81it/s]
 78%|███████▊  | 1204/1545 [10:36<03:07,  1.82it/s]
 78%|███████▊  | 1205/1545 [10:36<02:55,  1.93it/s]
 78%|███████▊  | 1206/1545 [10:37<03:01,  1.87it/s]
 78%|███████▊  | 1207/1545 [10:37<03:03,  1.84it/s]
 78%|███████▊  | 1208/1545 [10:38<02:58,  1.89it/s]
 78%|███████▊  | 1209/1545 [10:39<03:00,  1.86it/s]
 78%|███████▊  | 1210/1545 [10:39<03:06,  1.80it/s]
                                                   
{'loss': 0.0022, 'grad_norm': 2.473825588822365e-10, 'learning_rate': 2.1682847896440134e-06, 'rewards/chosen': -17.865224838256836, 'rewards/rejected': -44.9857063293457, 'rewards/accuracies': 1.0, 'rewards/margins': 27.120479583740234, 'logps/chosen': -318.7569274902344, 'logps/rejected': -567.396240234375, 'logits/chosen': -2.13226580619812, 'logits/rejected': -3.3080012798309326, 'epoch': 0.78}

 78%|███████▊  | 1210/1545 [10:39<03:06,  1.80it/s]
 78%|███████▊  | 1211/1545 [10:40<03:06,  1.79it/s]
 78%|███████▊  | 1212/1545 [10:40<02:54,  1.91it/s]
 79%|███████▊  | 1213/1545 [10:41<03:02,  1.82it/s]
 79%|███████▊  | 1214/1545 [10:41<02:42,  2.04it/s]
 79%|███████▊  | 1215/1545 [10:41<02:30,  2.20it/s]
 79%|███████▊  | 1216/1545 [10:42<02:28,  2.21it/s]
 79%|███████▉  | 1217/1545 [10:43<02:42,  2.02it/s]
 79%|███████▉  | 1218/1545 [10:43<02:49,  1.93it/s]
 79%|███████▉  | 1219/1545 [10:44<02:53,  1.88it/s]
 79%|███████▉  | 1220/1545 [10:44<02:51,  1.90it/s]
                                                   
{'loss': 1.6899, 'grad_norm': 1152.0, 'learning_rate': 2.103559870550162e-06, 'rewards/chosen': -28.565698623657227, 'rewards/rejected': -47.352169036865234, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 18.78647232055664, 'logps/chosen': -429.1759338378906, 'logps/rejected': -576.1998901367188, 'logits/chosen': -2.689363718032837, 'logits/rejected': -3.7332401275634766, 'epoch': 0.79}

 79%|███████▉  | 1220/1545 [10:44<02:51,  1.90it/s]
 79%|███████▉  | 1221/1545 [10:45<02:57,  1.83it/s]
 79%|███████▉  | 1222/1545 [10:45<03:00,  1.79it/s]
 79%|███████▉  | 1223/1545 [10:46<02:47,  1.92it/s]
 79%|███████▉  | 1224/1545 [10:46<02:55,  1.83it/s]
 79%|███████▉  | 1225/1545 [10:47<02:55,  1.82it/s]
 79%|███████▉  | 1226/1545 [10:47<02:54,  1.83it/s]
 79%|███████▉  | 1227/1545 [10:48<02:53,  1.83it/s]
 79%|███████▉  | 1228/1545 [10:49<02:54,  1.82it/s]
 80%|███████▉  | 1229/1545 [10:50<04:09,  1.26it/s]
 80%|███████▉  | 1230/1545 [10:50<03:27,  1.52it/s]
                                                   
{'loss': 0.8983, 'grad_norm': 2.4400651454925537e-07, 'learning_rate': 2.0388349514563107e-06, 'rewards/chosen': -17.619848251342773, 'rewards/rejected': -43.817138671875, 'rewards/accuracies': 0.699999988079071, 'rewards/margins': 26.19728660583496, 'logps/chosen': -310.8443298339844, 'logps/rejected': -550.3328857421875, 'logits/chosen': -2.216456651687622, 'logits/rejected': -3.418893814086914, 'epoch': 0.8}

 80%|███████▉  | 1230/1545 [10:50<03:27,  1.52it/s]
 80%|███████▉  | 1231/1545 [10:51<03:16,  1.60it/s]
 80%|███████▉  | 1232/1545 [10:52<03:20,  1.56it/s]
 80%|███████▉  | 1233/1545 [10:52<03:16,  1.59it/s]
 80%|███████▉  | 1234/1545 [10:53<02:59,  1.73it/s]
 80%|███████▉  | 1235/1545 [10:53<03:00,  1.72it/s]
 80%|████████  | 1236/1545 [10:54<02:57,  1.74it/s]
 80%|████████  | 1237/1545 [10:54<02:53,  1.77it/s]
 80%|████████  | 1238/1545 [10:55<02:52,  1.78it/s]
 80%|████████  | 1239/1545 [10:55<02:55,  1.74it/s]
 80%|████████  | 1240/1545 [10:56<02:54,  1.75it/s]
                                                   
{'loss': 1.4954, 'grad_norm': 1.7394086171407253e-11, 'learning_rate': 1.9741100323624596e-06, 'rewards/chosen': -20.375286102294922, 'rewards/rejected': -44.56410217285156, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 24.18881607055664, 'logps/chosen': -332.5040283203125, 'logps/rejected': -539.218505859375, 'logits/chosen': -2.468999147415161, 'logits/rejected': -3.7816715240478516, 'epoch': 0.8}

 80%|████████  | 1240/1545 [10:56<02:54,  1.75it/s]
 80%|████████  | 1241/1545 [10:56<02:43,  1.86it/s]
 80%|████████  | 1242/1545 [10:57<02:28,  2.04it/s]
 80%|████████  | 1243/1545 [10:57<02:37,  1.91it/s]
 81%|████████  | 1244/1545 [10:58<02:40,  1.88it/s]
 81%|████████  | 1245/1545 [10:58<02:31,  1.98it/s]
 81%|████████  | 1246/1545 [10:59<02:37,  1.90it/s]
 81%|████████  | 1247/1545 [11:00<02:42,  1.83it/s]
 81%|████████  | 1248/1545 [11:00<02:38,  1.88it/s]
 81%|████████  | 1249/1545 [11:01<02:42,  1.82it/s]
 81%|████████  | 1250/1545 [11:01<02:45,  1.79it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.0177001953125, 'learning_rate': 1.9093851132686085e-06, 'rewards/chosen': -10.88851261138916, 'rewards/rejected': -38.96172332763672, 'rewards/accuracies': 1.0, 'rewards/margins': 28.07320785522461, 'logps/chosen': -251.8507537841797, 'logps/rejected': -504.0205993652344, 'logits/chosen': -1.7540420293807983, 'logits/rejected': -3.0294809341430664, 'epoch': 0.81}

 81%|████████  | 1250/1545 [11:01<02:45,  1.79it/s]
 81%|████████  | 1251/1545 [11:02<02:46,  1.76it/s]
 81%|████████  | 1252/1545 [11:02<02:42,  1.80it/s]
 81%|████████  | 1253/1545 [11:03<02:45,  1.76it/s]
 81%|████████  | 1254/1545 [11:04<02:46,  1.75it/s]
 81%|████████  | 1255/1545 [11:04<02:33,  1.89it/s]
 81%|████████▏ | 1256/1545 [11:05<02:39,  1.82it/s]
 81%|████████▏ | 1257/1545 [11:05<02:39,  1.81it/s]
 81%|████████▏ | 1258/1545 [11:06<02:38,  1.81it/s]
 81%|████████▏ | 1259/1545 [11:06<02:35,  1.84it/s]
 82%|████████▏ | 1260/1545 [11:07<02:38,  1.80it/s]
                                                   
{'loss': 0.0, 'grad_norm': 8.96453857421875e-05, 'learning_rate': 1.8446601941747574e-06, 'rewards/chosen': -16.425739288330078, 'rewards/rejected': -49.47471237182617, 'rewards/accuracies': 1.0, 'rewards/margins': 33.04896926879883, 'logps/chosen': -337.70025634765625, 'logps/rejected': -628.4820556640625, 'logits/chosen': -1.5739221572875977, 'logits/rejected': -3.0157792568206787, 'epoch': 0.82}

 82%|████████▏ | 1260/1545 [11:07<02:38,  1.80it/s]
 82%|████████▏ | 1261/1545 [11:07<02:41,  1.76it/s]
 82%|████████▏ | 1262/1545 [11:08<02:20,  2.02it/s]
 82%|████████▏ | 1263/1545 [11:08<02:26,  1.93it/s]
 82%|████████▏ | 1264/1545 [11:09<02:30,  1.87it/s]
 82%|████████▏ | 1265/1545 [11:09<02:31,  1.84it/s]
 82%|████████▏ | 1266/1545 [11:10<02:22,  1.95it/s]
 82%|████████▏ | 1267/1545 [11:10<02:29,  1.86it/s]
 82%|████████▏ | 1268/1545 [11:11<02:30,  1.84it/s]
 82%|████████▏ | 1269/1545 [11:12<02:27,  1.87it/s]
 82%|████████▏ | 1270/1545 [11:12<02:29,  1.84it/s]
                                                   
{'loss': 0.2319, 'grad_norm': 1.3096723705530167e-10, 'learning_rate': 1.7799352750809063e-06, 'rewards/chosen': -19.763113021850586, 'rewards/rejected': -38.63679504394531, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 18.873680114746094, 'logps/chosen': -342.18231201171875, 'logps/rejected': -496.32232666015625, 'logits/chosen': -2.013960361480713, 'logits/rejected': -2.9975972175598145, 'epoch': 0.82}

 82%|████████▏ | 1270/1545 [11:12<02:29,  1.84it/s]
 82%|████████▏ | 1271/1545 [11:13<02:32,  1.80it/s]
 82%|████████▏ | 1272/1545 [11:13<02:33,  1.78it/s]
 82%|████████▏ | 1273/1545 [11:14<02:21,  1.92it/s]
 82%|████████▏ | 1274/1545 [11:14<02:25,  1.86it/s]
 83%|████████▎ | 1275/1545 [11:15<02:27,  1.84it/s]
 83%|████████▎ | 1276/1545 [11:15<02:24,  1.86it/s]
 83%|████████▎ | 1277/1545 [11:16<02:25,  1.84it/s]
 83%|████████▎ | 1278/1545 [11:17<02:29,  1.78it/s]
 83%|████████▎ | 1279/1545 [11:17<02:29,  1.77it/s]
 83%|████████▎ | 1280/1545 [11:17<02:17,  1.93it/s]
                                                   
{'loss': 0.444, 'grad_norm': 7.771561172376096e-16, 'learning_rate': 1.715210355987055e-06, 'rewards/chosen': -16.24677085876465, 'rewards/rejected': -37.7825927734375, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 21.535823822021484, 'logps/chosen': -301.96905517578125, 'logps/rejected': -496.69610595703125, 'logits/chosen': -1.7921947240829468, 'logits/rejected': -2.7485709190368652, 'epoch': 0.83}

 83%|████████▎ | 1280/1545 [11:18<02:17,  1.93it/s]
 83%|████████▎ | 1281/1545 [11:18<02:24,  1.83it/s]
 83%|████████▎ | 1282/1545 [11:19<02:23,  1.83it/s]
 83%|████████▎ | 1283/1545 [11:19<02:20,  1.87it/s]
 83%|████████▎ | 1284/1545 [11:20<02:17,  1.89it/s]
 83%|████████▎ | 1285/1545 [11:20<02:21,  1.84it/s]
 83%|████████▎ | 1286/1545 [11:21<02:21,  1.83it/s]
 83%|████████▎ | 1287/1545 [11:21<02:12,  1.94it/s]
 83%|████████▎ | 1288/1545 [11:22<02:01,  2.11it/s]
 83%|████████▎ | 1289/1545 [11:22<02:08,  2.00it/s]
 83%|████████▎ | 1290/1545 [11:23<02:11,  1.93it/s]
                                                   
{'loss': 0.066, 'grad_norm': 6.668269634246826e-07, 'learning_rate': 1.650485436893204e-06, 'rewards/chosen': -12.632109642028809, 'rewards/rejected': -42.14348602294922, 'rewards/accuracies': 1.0, 'rewards/margins': 29.511377334594727, 'logps/chosen': -311.40838623046875, 'logps/rejected': -553.1618041992188, 'logits/chosen': -1.4785627126693726, 'logits/rejected': -2.7886316776275635, 'epoch': 0.83}

 83%|████████▎ | 1290/1545 [11:23<02:11,  1.93it/s]
 84%|████████▎ | 1291/1545 [11:23<02:08,  1.97it/s]
 84%|████████▎ | 1292/1545 [11:24<02:13,  1.89it/s]
 84%|████████▎ | 1293/1545 [11:24<02:17,  1.83it/s]
 84%|████████▍ | 1294/1545 [11:25<02:16,  1.84it/s]
 84%|████████▍ | 1295/1545 [11:25<02:16,  1.83it/s]
 84%|████████▍ | 1296/1545 [11:26<02:18,  1.79it/s]
 84%|████████▍ | 1297/1545 [11:27<02:19,  1.78it/s]
 84%|████████▍ | 1298/1545 [11:27<02:07,  1.94it/s]
 84%|████████▍ | 1299/1545 [11:28<02:13,  1.84it/s]
 84%|████████▍ | 1300/1545 [11:28<02:13,  1.84it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.01336669921875, 'learning_rate': 1.585760517799353e-06, 'rewards/chosen': -15.508363723754883, 'rewards/rejected': -44.09370803833008, 'rewards/accuracies': 1.0, 'rewards/margins': 28.585346221923828, 'logps/chosen': -305.378662109375, 'logps/rejected': -551.26708984375, 'logits/chosen': -1.9857141971588135, 'logits/rejected': -2.942575454711914, 'epoch': 0.84}

 84%|████████▍ | 1300/1545 [11:28<02:13,  1.84it/s]
 84%|████████▍ | 1301/1545 [11:29<02:12,  1.84it/s]
 84%|████████▍ | 1302/1545 [11:29<02:10,  1.86it/s]
 84%|████████▍ | 1303/1545 [11:30<02:12,  1.82it/s]
 84%|████████▍ | 1304/1545 [11:30<02:14,  1.79it/s]
 84%|████████▍ | 1305/1545 [11:31<02:06,  1.90it/s]
 85%|████████▍ | 1306/1545 [11:31<02:11,  1.82it/s]
 85%|████████▍ | 1307/1545 [11:32<02:10,  1.82it/s]
 85%|████████▍ | 1308/1545 [11:33<02:09,  1.82it/s]
 85%|████████▍ | 1309/1545 [11:33<02:07,  1.86it/s]
 85%|████████▍ | 1310/1545 [11:34<02:09,  1.82it/s]
                                                   
{'loss': 0.2831, 'grad_norm': 5.438923835754395e-07, 'learning_rate': 1.5210355987055017e-06, 'rewards/chosen': -18.139490127563477, 'rewards/rejected': -49.077415466308594, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 30.93792724609375, 'logps/chosen': -349.8685607910156, 'logps/rejected': -612.5897216796875, 'logits/chosen': -1.8611408472061157, 'logits/rejected': -3.1098215579986572, 'epoch': 0.85}

 85%|████████▍ | 1310/1545 [11:34<02:09,  1.82it/s]
 85%|████████▍ | 1311/1545 [11:34<02:10,  1.80it/s]
 85%|████████▍ | 1312/1545 [11:35<02:01,  1.92it/s]
 85%|████████▍ | 1313/1545 [11:35<02:04,  1.86it/s]
 85%|████████▌ | 1314/1545 [11:36<01:52,  2.05it/s]
 85%|████████▌ | 1315/1545 [11:36<01:58,  1.95it/s]
 85%|████████▌ | 1316/1545 [11:37<01:51,  2.05it/s]
 85%|████████▌ | 1317/1545 [11:37<01:56,  1.95it/s]
 85%|████████▌ | 1318/1545 [11:38<01:46,  2.12it/s]
 85%|████████▌ | 1319/1545 [11:38<01:54,  1.98it/s]
 85%|████████▌ | 1320/1545 [11:39<01:49,  2.05it/s]
                                                   
{'loss': 0.0, 'grad_norm': 9.75781955236954e-17, 'learning_rate': 1.4563106796116506e-06, 'rewards/chosen': -10.908350944519043, 'rewards/rejected': -47.042503356933594, 'rewards/accuracies': 1.0, 'rewards/margins': 36.134151458740234, 'logps/chosen': -251.7775421142578, 'logps/rejected': -576.4340209960938, 'logits/chosen': -1.4616104364395142, 'logits/rejected': -3.171027898788452, 'epoch': 0.85}

 85%|████████▌ | 1320/1545 [11:39<01:49,  2.05it/s]
 86%|████████▌ | 1321/1545 [11:39<01:57,  1.91it/s]
 86%|████████▌ | 1322/1545 [11:40<01:58,  1.87it/s]
 86%|████████▌ | 1323/1545 [11:40<01:59,  1.86it/s]
 86%|████████▌ | 1324/1545 [11:41<01:57,  1.88it/s]
 86%|████████▌ | 1325/1545 [11:41<01:59,  1.84it/s]
 86%|████████▌ | 1326/1545 [11:42<02:00,  1.81it/s]
 86%|████████▌ | 1327/1545 [11:42<01:44,  2.08it/s]
 86%|████████▌ | 1328/1545 [11:43<01:49,  1.97it/s]
 86%|████████▌ | 1329/1545 [11:43<01:53,  1.90it/s]
 86%|████████▌ | 1330/1545 [11:44<01:54,  1.88it/s]
                                                   
{'loss': 0.0001, 'grad_norm': 2.34375, 'learning_rate': 1.3915857605177997e-06, 'rewards/chosen': -12.354753494262695, 'rewards/rejected': -41.89970016479492, 'rewards/accuracies': 1.0, 'rewards/margins': 29.54494857788086, 'logps/chosen': -297.7950134277344, 'logps/rejected': -539.6881103515625, 'logits/chosen': -1.1807644367218018, 'logits/rejected': -2.6145424842834473, 'epoch': 0.86}

 86%|████████▌ | 1330/1545 [11:44<01:54,  1.88it/s]
 86%|████████▌ | 1331/1545 [11:44<01:47,  2.00it/s]
 86%|████████▌ | 1332/1545 [11:45<01:38,  2.15it/s]
 86%|████████▋ | 1333/1545 [11:45<01:45,  2.02it/s]
 86%|████████▋ | 1334/1545 [11:46<01:49,  1.93it/s]
 86%|████████▋ | 1335/1545 [11:46<01:42,  2.05it/s]
 86%|████████▋ | 1336/1545 [11:47<01:48,  1.93it/s]
 87%|████████▋ | 1337/1545 [11:47<01:50,  1.88it/s]
 87%|████████▋ | 1338/1545 [11:48<01:49,  1.89it/s]
 87%|████████▋ | 1339/1545 [11:49<01:47,  1.91it/s]
 87%|████████▋ | 1340/1545 [11:49<01:50,  1.85it/s]
                                                   
{'loss': 0.0001, 'grad_norm': 1.2747477740049362e-08, 'learning_rate': 1.3268608414239483e-06, 'rewards/chosen': -7.87436580657959, 'rewards/rejected': -39.97648239135742, 'rewards/accuracies': 1.0, 'rewards/margins': 32.10211944580078, 'logps/chosen': -211.8810577392578, 'logps/rejected': -505.80975341796875, 'logits/chosen': -1.697493314743042, 'logits/rejected': -3.0915396213531494, 'epoch': 0.87}

 87%|████████▋ | 1340/1545 [11:49<01:50,  1.85it/s]
 87%|████████▋ | 1341/1545 [11:50<01:52,  1.82it/s]
 87%|████████▋ | 1342/1545 [11:50<01:46,  1.90it/s]
 87%|████████▋ | 1343/1545 [11:52<02:47,  1.20it/s]
 87%|████████▋ | 1344/1545 [11:52<02:30,  1.33it/s]
 87%|████████▋ | 1345/1545 [11:53<02:16,  1.47it/s]
 87%|████████▋ | 1346/1545 [11:53<02:09,  1.53it/s]
 87%|████████▋ | 1347/1545 [11:54<02:05,  1.57it/s]
 87%|████████▋ | 1348/1545 [11:54<02:00,  1.64it/s]
 87%|████████▋ | 1349/1545 [11:55<01:48,  1.80it/s]
 87%|████████▋ | 1350/1545 [11:56<01:49,  1.78it/s]
                                                   
{'loss': 0.0, 'grad_norm': 5.327165126800537e-07, 'learning_rate': 1.2621359223300972e-06, 'rewards/chosen': -13.375717163085938, 'rewards/rejected': -49.55775833129883, 'rewards/accuracies': 1.0, 'rewards/margins': 36.18204116821289, 'logps/chosen': -304.5226135253906, 'logps/rejected': -614.50341796875, 'logits/chosen': -1.5832102298736572, 'logits/rejected': -3.0017504692077637, 'epoch': 0.87}

 87%|████████▋ | 1350/1545 [11:56<01:49,  1.78it/s]
 87%|████████▋ | 1351/1545 [11:56<01:48,  1.78it/s]
 88%|████████▊ | 1352/1545 [11:57<01:44,  1.85it/s]
 88%|████████▊ | 1353/1545 [11:57<01:43,  1.86it/s]
 88%|████████▊ | 1354/1545 [11:58<01:45,  1.80it/s]
 88%|████████▊ | 1355/1545 [11:58<01:46,  1.78it/s]
 88%|████████▊ | 1356/1545 [11:59<01:39,  1.90it/s]
 88%|████████▊ | 1357/1545 [11:59<01:30,  2.07it/s]
 88%|████████▊ | 1358/1545 [12:00<01:36,  1.94it/s]
 88%|████████▊ | 1359/1545 [12:00<01:38,  1.88it/s]
 88%|████████▊ | 1360/1545 [12:01<01:30,  2.04it/s]
                                                   
{'loss': 0.0, 'grad_norm': 0.0023956298828125, 'learning_rate': 1.197411003236246e-06, 'rewards/chosen': -8.20715045928955, 'rewards/rejected': -40.68087387084961, 'rewards/accuracies': 1.0, 'rewards/margins': 32.47372817993164, 'logps/chosen': -225.0033416748047, 'logps/rejected': -529.2271118164062, 'logits/chosen': -1.879974603652954, 'logits/rejected': -2.8071045875549316, 'epoch': 0.88}

 88%|████████▊ | 1360/1545 [12:01<01:30,  2.04it/s]
 88%|████████▊ | 1361/1545 [12:01<01:35,  1.93it/s]
 88%|████████▊ | 1362/1545 [12:02<01:26,  2.12it/s]
 88%|████████▊ | 1363/1545 [12:02<01:31,  1.99it/s]
 88%|████████▊ | 1364/1545 [12:03<01:26,  2.10it/s]
 88%|████████▊ | 1365/1545 [12:03<01:32,  1.94it/s]
 88%|████████▊ | 1366/1545 [12:04<01:34,  1.90it/s]
 88%|████████▊ | 1367/1545 [12:04<01:33,  1.91it/s]
 89%|████████▊ | 1368/1545 [12:05<01:33,  1.89it/s]
 89%|████████▊ | 1369/1545 [12:05<01:35,  1.85it/s]
 89%|████████▊ | 1370/1545 [12:06<01:36,  1.81it/s]
                                                   
{'loss': 0.0, 'grad_norm': 3.245077095925808e-09, 'learning_rate': 1.132686084142395e-06, 'rewards/chosen': -13.201225280761719, 'rewards/rejected': -47.207088470458984, 'rewards/accuracies': 1.0, 'rewards/margins': 34.005863189697266, 'logps/chosen': -289.7091369628906, 'logps/rejected': -593.4539794921875, 'logits/chosen': -1.5826839208602905, 'logits/rejected': -2.911431074142456, 'epoch': 0.89}

 89%|████████▊ | 1370/1545 [12:06<01:36,  1.81it/s]
 89%|████████▊ | 1371/1545 [12:06<01:30,  1.92it/s]
 89%|████████▉ | 1372/1545 [12:07<01:33,  1.85it/s]
 89%|████████▉ | 1373/1545 [12:08<01:33,  1.84it/s]
 89%|████████▉ | 1374/1545 [12:08<01:32,  1.85it/s]
 89%|████████▉ | 1375/1545 [12:09<01:30,  1.88it/s]
 89%|████████▉ | 1376/1545 [12:09<01:32,  1.83it/s]
 89%|████████▉ | 1377/1545 [12:10<01:31,  1.83it/s]
 89%|████████▉ | 1378/1545 [12:10<01:25,  1.96it/s]
 89%|████████▉ | 1379/1545 [12:11<01:28,  1.87it/s]
 89%|████████▉ | 1380/1545 [12:11<01:29,  1.84it/s]
                                                   
{'loss': 0.0046, 'grad_norm': 2.625, 'learning_rate': 1.0679611650485437e-06, 'rewards/chosen': -19.794082641601562, 'rewards/rejected': -42.44209671020508, 'rewards/accuracies': 1.0, 'rewards/margins': 22.648014068603516, 'logps/chosen': -388.0662841796875, 'logps/rejected': -544.12451171875, 'logits/chosen': -1.7488387823104858, 'logits/rejected': -2.9612879753112793, 'epoch': 0.89}

 89%|████████▉ | 1380/1545 [12:11<01:29,  1.84it/s]
 89%|████████▉ | 1381/1545 [12:12<01:28,  1.86it/s]
 89%|████████▉ | 1382/1545 [12:12<01:21,  2.00it/s]
 90%|████████▉ | 1383/1545 [12:13<01:25,  1.89it/s]
 90%|████████▉ | 1384/1545 [12:13<01:27,  1.84it/s]
 90%|████████▉ | 1385/1545 [12:14<01:24,  1.90it/s]
 90%|████████▉ | 1386/1545 [12:14<01:16,  2.09it/s]
 90%|████████▉ | 1387/1545 [12:15<01:20,  1.95it/s]
 90%|████████▉ | 1388/1545 [12:15<01:22,  1.90it/s]
 90%|████████▉ | 1389/1545 [12:16<01:19,  1.96it/s]
 90%|████████▉ | 1390/1545 [12:16<01:22,  1.88it/s]
                                                   
{'loss': 0.0058, 'grad_norm': 75.5, 'learning_rate': 1.0032362459546926e-06, 'rewards/chosen': -14.03381633758545, 'rewards/rejected': -39.99800109863281, 'rewards/accuracies': 1.0, 'rewards/margins': 25.964187622070312, 'logps/chosen': -306.3893127441406, 'logps/rejected': -523.348388671875, 'logits/chosen': -1.803492784500122, 'logits/rejected': -2.9958126544952393, 'epoch': 0.9}

 90%|████████▉ | 1390/1545 [12:16<01:22,  1.88it/s]
 90%|█████████ | 1391/1545 [12:17<01:23,  1.84it/s]
 90%|█████████ | 1392/1545 [12:18<01:22,  1.85it/s]
 90%|█████████ | 1393/1545 [12:18<01:16,  1.98it/s]
 90%|█████████ | 1394/1545 [12:19<01:21,  1.86it/s]
 90%|█████████ | 1395/1545 [12:19<01:21,  1.84it/s]
 90%|█████████ | 1396/1545 [12:20<01:17,  1.91it/s]
 90%|█████████ | 1397/1545 [12:20<01:18,  1.89it/s]
 90%|█████████ | 1398/1545 [12:21<01:19,  1.85it/s]
 91%|█████████ | 1399/1545 [12:21<01:19,  1.85it/s]
 91%|█████████ | 1400/1545 [12:22<01:13,  1.98it/s]
                                                   
{'loss': 0.0, 'grad_norm': 4.298783551348606e-13, 'learning_rate': 9.385113268608415e-07, 'rewards/chosen': -17.3962345123291, 'rewards/rejected': -49.274452209472656, 'rewards/accuracies': 1.0, 'rewards/margins': 31.878215789794922, 'logps/chosen': -298.85546875, 'logps/rejected': -580.9243774414062, 'logits/chosen': -1.9781240224838257, 'logits/rejected': -3.3585548400878906, 'epoch': 0.91}

 91%|█████████ | 1400/1545 [12:22<01:13,  1.98it/s]
 91%|█████████ | 1401/1545 [12:22<01:15,  1.90it/s]
 91%|█████████ | 1402/1545 [12:23<01:16,  1.88it/s]
 91%|█████████ | 1403/1545 [12:23<01:15,  1.88it/s]
 91%|█████████ | 1404/1545 [12:24<01:12,  1.95it/s]
 91%|█████████ | 1405/1545 [12:24<01:06,  2.11it/s]
 91%|█████████ | 1406/1545 [12:25<01:09,  1.99it/s]
 91%|█████████ | 1407/1545 [12:25<01:10,  1.95it/s]
 91%|█████████ | 1408/1545 [12:26<01:10,  1.93it/s]
 91%|█████████ | 1409/1545 [12:26<01:13,  1.86it/s]
 91%|█████████▏| 1410/1545 [12:27<01:13,  1.83it/s]
                                                   
{'loss': 0.0, 'grad_norm': 1.4137996329210978e-16, 'learning_rate': 8.737864077669904e-07, 'rewards/chosen': -10.19818115234375, 'rewards/rejected': -47.04527282714844, 'rewards/accuracies': 1.0, 'rewards/margins': 36.847084045410156, 'logps/chosen': -266.72540283203125, 'logps/rejected': -583.9825439453125, 'logits/chosen': -1.446902871131897, 'logits/rejected': -3.2011420726776123, 'epoch': 0.91}

 91%|█████████▏| 1410/1545 [12:27<01:13,  1.83it/s]
 91%|█████████▏| 1411/1545 [12:27<01:10,  1.89it/s]
 91%|█████████▏| 1412/1545 [12:28<01:12,  1.84it/s]
 91%|█████████▏| 1413/1545 [12:29<01:12,  1.82it/s]
 92%|█████████▏| 1414/1545 [12:29<01:11,  1.83it/s]
 92%|█████████▏| 1415/1545 [12:30<01:09,  1.86it/s]
 92%|█████████▏| 1416/1545 [12:30<01:12,  1.78it/s]
 92%|█████████▏| 1417/1545 [12:31<01:11,  1.79it/s]
 92%|█████████▏| 1418/1545 [12:31<01:06,  1.90it/s]
 92%|█████████▏| 1419/1545 [12:32<01:08,  1.85it/s]
 92%|█████████▏| 1420/1545 [12:32<01:08,  1.82it/s]
                                                   
{'loss': 0.0013, 'grad_norm': 1.0244548320770264e-07, 'learning_rate': 8.090614886731392e-07, 'rewards/chosen': -20.795812606811523, 'rewards/rejected': -50.17866897583008, 'rewards/accuracies': 1.0, 'rewards/margins': 29.38285255432129, 'logps/chosen': -362.2292785644531, 'logps/rejected': -605.4534301757812, 'logits/chosen': -2.1136467456817627, 'logits/rejected': -3.451559543609619, 'epoch': 0.92}

 92%|█████████▏| 1420/1545 [12:32<01:08,  1.82it/s]
 92%|█████████▏| 1421/1545 [12:33<01:08,  1.82it/s]
 92%|█████████▏| 1422/1545 [12:34<01:06,  1.85it/s]
 92%|█████████▏| 1423/1545 [12:34<01:07,  1.81it/s]
 92%|█████████▏| 1424/1545 [12:35<01:07,  1.79it/s]
 92%|█████████▏| 1425/1545 [12:35<01:03,  1.90it/s]
 92%|█████████▏| 1426/1545 [12:36<01:05,  1.82it/s]
 92%|█████████▏| 1427/1545 [12:36<01:04,  1.82it/s]
 92%|█████████▏| 1428/1545 [12:37<01:04,  1.81it/s]
 92%|█████████▏| 1429/1545 [12:37<01:02,  1.85it/s]
 93%|█████████▎| 1430/1545 [12:38<01:04,  1.80it/s]
                                                   
{'loss': 0.0001, 'grad_norm': 7.486343383789062e-05, 'learning_rate': 7.443365695792882e-07, 'rewards/chosen': -17.025291442871094, 'rewards/rejected': -57.489990234375, 'rewards/accuracies': 1.0, 'rewards/margins': 40.46469497680664, 'logps/chosen': -336.61431884765625, 'logps/rejected': -702.4564208984375, 'logits/chosen': -1.5203006267547607, 'logits/rejected': -3.2374939918518066, 'epoch': 0.93}

 93%|█████████▎| 1430/1545 [12:38<01:04,  1.80it/s]
 93%|█████████▎| 1431/1545 [12:39<01:04,  1.78it/s]
 93%|█████████▎| 1432/1545 [12:39<00:59,  1.89it/s]
 93%|█████████▎| 1433/1545 [12:40<01:01,  1.82it/s]
 93%|█████████▎| 1434/1545 [12:40<01:01,  1.80it/s]
 93%|█████████▎| 1435/1545 [12:41<01:01,  1.79it/s]
 93%|█████████▎| 1436/1545 [12:41<00:58,  1.86it/s]
 93%|█████████▎| 1437/1545 [12:42<00:59,  1.82it/s]
 93%|█████████▎| 1438/1545 [12:42<00:59,  1.80it/s]
 93%|█████████▎| 1439/1545 [12:43<00:54,  1.93it/s]
 93%|█████████▎| 1440/1545 [12:43<00:57,  1.84it/s]
                                                   
{'loss': 0.0775, 'grad_norm': 3.4421682357788086e-06, 'learning_rate': 6.79611650485437e-07, 'rewards/chosen': -16.205768585205078, 'rewards/rejected': -38.98480224609375, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 22.779033660888672, 'logps/chosen': -307.8979797363281, 'logps/rejected': -502.49041748046875, 'logits/chosen': -2.063363552093506, 'logits/rejected': -3.2290992736816406, 'epoch': 0.93}

 93%|█████████▎| 1440/1545 [12:43<00:57,  1.84it/s]
 93%|█████████▎| 1441/1545 [12:44<00:58,  1.79it/s]
 93%|█████████▎| 1442/1545 [12:44<00:56,  1.81it/s]
 93%|█████████▎| 1443/1545 [12:45<00:55,  1.85it/s]
 93%|█████████▎| 1444/1545 [12:46<00:55,  1.83it/s]
 94%|█████████▎| 1445/1545 [12:46<00:55,  1.81it/s]
 94%|█████████▎| 1446/1545 [12:47<00:51,  1.93it/s]
 94%|█████████▎| 1447/1545 [12:47<00:52,  1.87it/s]
 94%|█████████▎| 1448/1545 [12:48<00:52,  1.83it/s]
 94%|█████████▍| 1449/1545 [12:48<00:53,  1.78it/s]
 94%|█████████▍| 1450/1545 [12:49<00:51,  1.84it/s]
                                                   
{'loss': 0.0001, 'grad_norm': 1.2168044349891716e-13, 'learning_rate': 6.148867313915858e-07, 'rewards/chosen': -18.26565933227539, 'rewards/rejected': -48.65673828125, 'rewards/accuracies': 1.0, 'rewards/margins': 30.391077041625977, 'logps/chosen': -316.24761962890625, 'logps/rejected': -603.7527465820312, 'logits/chosen': -2.136662006378174, 'logits/rejected': -3.263352870941162, 'epoch': 0.94}

 94%|█████████▍| 1450/1545 [12:49<00:51,  1.84it/s]
 94%|█████████▍| 1451/1545 [12:49<00:53,  1.75it/s]
 94%|█████████▍| 1452/1545 [12:50<00:53,  1.74it/s]
 94%|█████████▍| 1453/1545 [12:50<00:49,  1.85it/s]
 94%|█████████▍| 1454/1545 [12:51<00:51,  1.77it/s]
 94%|█████████▍| 1455/1545 [12:52<00:50,  1.78it/s]
 94%|█████████▍| 1456/1545 [12:53<01:13,  1.21it/s]
 94%|█████████▍| 1457/1545 [12:54<01:06,  1.32it/s]
 94%|█████████▍| 1458/1545 [12:54<00:56,  1.55it/s]
 94%|█████████▍| 1459/1545 [12:55<00:53,  1.60it/s]
 94%|█████████▍| 1460/1545 [12:55<00:48,  1.75it/s]
                                                   
{'loss': 0.0, 'grad_norm': 2.0469737016526324e-16, 'learning_rate': 5.501618122977346e-07, 'rewards/chosen': -17.234251022338867, 'rewards/rejected': -46.855690002441406, 'rewards/accuracies': 1.0, 'rewards/margins': 29.621444702148438, 'logps/chosen': -348.11370849609375, 'logps/rejected': -579.3179321289062, 'logits/chosen': -1.4622868299484253, 'logits/rejected': -3.198202133178711, 'epoch': 0.94}

 94%|█████████▍| 1460/1545 [12:55<00:48,  1.75it/s]
 95%|█████████▍| 1461/1545 [12:56<00:49,  1.69it/s]
 95%|█████████▍| 1462/1545 [12:56<00:49,  1.69it/s]
 95%|█████████▍| 1463/1545 [12:57<00:47,  1.72it/s]
 95%|█████████▍| 1464/1545 [12:57<00:45,  1.77it/s]
 95%|█████████▍| 1465/1545 [12:58<00:45,  1.75it/s]
 95%|█████████▍| 1466/1545 [12:59<00:44,  1.76it/s]
 95%|█████████▍| 1467/1545 [12:59<00:41,  1.89it/s]
 95%|█████████▌| 1468/1545 [13:00<00:42,  1.81it/s]
 95%|█████████▌| 1469/1545 [13:00<00:41,  1.81it/s]
 95%|█████████▌| 1470/1545 [13:01<00:40,  1.83it/s]
                                                   
{'loss': 0.0, 'grad_norm': 1.4921397450962104e-13, 'learning_rate': 4.854368932038835e-07, 'rewards/chosen': -14.240861892700195, 'rewards/rejected': -46.95703887939453, 'rewards/accuracies': 1.0, 'rewards/margins': 32.7161750793457, 'logps/chosen': -306.51873779296875, 'logps/rejected': -581.2691040039062, 'logits/chosen': -1.777173638343811, 'logits/rejected': -3.262760877609253, 'epoch': 0.95}

 95%|█████████▌| 1470/1545 [13:01<00:40,  1.83it/s]
 95%|█████████▌| 1471/1545 [13:01<00:35,  2.08it/s]
 95%|█████████▌| 1472/1545 [13:02<00:37,  1.96it/s]
 95%|█████████▌| 1473/1545 [13:02<00:37,  1.91it/s]
 95%|█████████▌| 1474/1545 [13:03<00:36,  1.94it/s]
 95%|█████████▌| 1475/1545 [13:03<00:36,  1.91it/s]
 96%|█████████▌| 1476/1545 [13:04<00:37,  1.86it/s]
 96%|█████████▌| 1477/1545 [13:04<00:37,  1.81it/s]
 96%|█████████▌| 1478/1545 [13:05<00:34,  1.93it/s]
 96%|█████████▌| 1479/1545 [13:05<00:35,  1.86it/s]
 96%|█████████▌| 1480/1545 [13:06<00:35,  1.84it/s]
                                                   
{'loss': 0.001, 'grad_norm': 5.186961971048731e-13, 'learning_rate': 4.207119741100324e-07, 'rewards/chosen': -15.928776741027832, 'rewards/rejected': -49.93461608886719, 'rewards/accuracies': 1.0, 'rewards/margins': 34.00584030151367, 'logps/chosen': -290.5811767578125, 'logps/rejected': -614.055419921875, 'logits/chosen': -2.113398551940918, 'logits/rejected': -3.1592159271240234, 'epoch': 0.96}

 96%|█████████▌| 1480/1545 [13:06<00:35,  1.84it/s]
 96%|█████████▌| 1481/1545 [13:06<00:34,  1.83it/s]
 96%|█████████▌| 1482/1545 [13:07<00:34,  1.85it/s]
 96%|█████████▌| 1483/1545 [13:08<00:34,  1.82it/s]
 96%|█████████▌| 1484/1545 [13:08<00:33,  1.83it/s]
 96%|█████████▌| 1485/1545 [13:09<00:31,  1.93it/s]
 96%|█████████▌| 1486/1545 [13:09<00:31,  1.88it/s]
 96%|█████████▌| 1487/1545 [13:10<00:31,  1.84it/s]
 96%|█████████▋| 1488/1545 [13:10<00:31,  1.83it/s]
 96%|█████████▋| 1489/1545 [13:11<00:29,  1.87it/s]
 96%|█████████▋| 1490/1545 [13:11<00:30,  1.80it/s]
                                                   
{'loss': 0.3793, 'grad_norm': 8.836247705756861e-18, 'learning_rate': 3.5598705501618125e-07, 'rewards/chosen': -12.482555389404297, 'rewards/rejected': -39.87023162841797, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 27.387676239013672, 'logps/chosen': -264.73980712890625, 'logps/rejected': -496.36474609375, 'logits/chosen': -1.77366042137146, 'logits/rejected': -3.205986499786377, 'epoch': 0.96}

 96%|█████████▋| 1490/1545 [13:11<00:30,  1.80it/s]
 97%|█████████▋| 1491/1545 [13:12<00:30,  1.77it/s]
 97%|█████████▋| 1492/1545 [13:12<00:28,  1.89it/s]
 97%|█████████▋| 1493/1545 [13:13<00:28,  1.83it/s]
 97%|█████████▋| 1494/1545 [13:14<00:28,  1.82it/s]
 97%|█████████▋| 1495/1545 [13:14<00:27,  1.82it/s]
 97%|█████████▋| 1496/1545 [13:15<00:26,  1.84it/s]
 97%|█████████▋| 1497/1545 [13:15<00:26,  1.81it/s]
 97%|█████████▋| 1498/1545 [13:16<00:25,  1.81it/s]
 97%|█████████▋| 1499/1545 [13:16<00:23,  1.95it/s]
 97%|█████████▋| 1500/1545 [13:17<00:23,  1.89it/s]
                                                   
{'loss': 0.0, 'grad_norm': 4.729550084903167e-14, 'learning_rate': 2.9126213592233014e-07, 'rewards/chosen': -16.625436782836914, 'rewards/rejected': -45.96239471435547, 'rewards/accuracies': 1.0, 'rewards/margins': 29.336956024169922, 'logps/chosen': -287.1039733886719, 'logps/rejected': -579.9110107421875, 'logits/chosen': -2.09769868850708, 'logits/rejected': -3.411703109741211, 'epoch': 0.97}

 97%|█████████▋| 1500/1545 [13:17<00:23,  1.89it/s]
 97%|█████████▋| 1501/1545 [13:17<00:23,  1.85it/s]
 97%|█████████▋| 1502/1545 [13:18<00:21,  2.04it/s]
 97%|█████████▋| 1503/1545 [13:18<00:19,  2.12it/s]
 97%|█████████▋| 1504/1545 [13:19<00:20,  2.00it/s]
 97%|█████████▋| 1505/1545 [13:19<00:20,  1.94it/s]
 97%|█████████▋| 1506/1545 [13:20<00:20,  1.91it/s]
 98%|█████████▊| 1507/1545 [13:20<00:18,  2.04it/s]
 98%|█████████▊| 1508/1545 [13:21<00:19,  1.93it/s]
 98%|█████████▊| 1509/1545 [13:21<00:17,  2.11it/s]
 98%|█████████▊| 1510/1545 [13:22<00:17,  2.02it/s]
                                                   
{'loss': 0.0693, 'grad_norm': 4.041939973831177e-07, 'learning_rate': 2.26537216828479e-07, 'rewards/chosen': -13.022150039672852, 'rewards/rejected': -43.39824676513672, 'rewards/accuracies': 0.8999999761581421, 'rewards/margins': 30.3760929107666, 'logps/chosen': -266.9024963378906, 'logps/rejected': -551.7422485351562, 'logits/chosen': -1.764814019203186, 'logits/rejected': -3.2696170806884766, 'epoch': 0.98}

 98%|█████████▊| 1510/1545 [13:22<00:17,  2.02it/s]
 98%|█████████▊| 1511/1545 [13:22<00:16,  2.11it/s]
 98%|█████████▊| 1512/1545 [13:23<00:16,  1.94it/s]
 98%|█████████▊| 1513/1545 [13:23<00:16,  1.91it/s]
 98%|█████████▊| 1514/1545 [13:24<00:16,  1.92it/s]
 98%|█████████▊| 1515/1545 [13:24<00:15,  1.90it/s]
 98%|█████████▊| 1516/1545 [13:25<00:15,  1.86it/s]
 98%|█████████▊| 1517/1545 [13:25<00:15,  1.84it/s]
 98%|█████████▊| 1518/1545 [13:26<00:13,  2.02it/s]
 98%|█████████▊| 1519/1545 [13:26<00:11,  2.20it/s]
 98%|█████████▊| 1520/1545 [13:27<00:12,  2.02it/s]
                                                   
{'loss': 0.0007, 'grad_norm': 6.28125, 'learning_rate': 1.6181229773462782e-07, 'rewards/chosen': -14.050992965698242, 'rewards/rejected': -44.175750732421875, 'rewards/accuracies': 1.0, 'rewards/margins': 30.124755859375, 'logps/chosen': -316.6773376464844, 'logps/rejected': -552.1802978515625, 'logits/chosen': -1.1631816625595093, 'logits/rejected': -3.36537504196167, 'epoch': 0.98}

 98%|█████████▊| 1520/1545 [13:27<00:12,  2.02it/s]
 98%|█████████▊| 1521/1545 [13:27<00:12,  1.94it/s]
 99%|█████████▊| 1522/1545 [13:28<00:11,  1.93it/s]
 99%|█████████▊| 1523/1545 [13:28<00:11,  1.91it/s]
 99%|█████████▊| 1524/1545 [13:29<00:11,  1.85it/s]
 99%|█████████▊| 1525/1545 [13:29<00:09,  2.05it/s]
 99%|█████████▉| 1526/1545 [13:30<00:09,  2.09it/s]
 99%|█████████▉| 1527/1545 [13:30<00:09,  1.93it/s]
 99%|█████████▉| 1528/1545 [13:31<00:09,  1.87it/s]
 99%|█████████▉| 1529/1545 [13:32<00:08,  1.85it/s]
 99%|█████████▉| 1530/1545 [13:32<00:07,  1.97it/s]
                                                   
{'loss': 0.0, 'grad_norm': 6.714628852932947e-13, 'learning_rate': 9.70873786407767e-08, 'rewards/chosen': -18.606670379638672, 'rewards/rejected': -48.73549270629883, 'rewards/accuracies': 1.0, 'rewards/margins': 30.12882423400879, 'logps/chosen': -354.6220703125, 'logps/rejected': -609.2911376953125, 'logits/chosen': -1.5294870138168335, 'logits/rejected': -3.3688862323760986, 'epoch': 0.99}

 99%|█████████▉| 1530/1545 [13:32<00:07,  1.97it/s]
 99%|█████████▉| 1531/1545 [13:33<00:07,  1.88it/s]
 99%|█████████▉| 1532/1545 [13:33<00:06,  1.86it/s]
 99%|█████████▉| 1533/1545 [13:33<00:05,  2.06it/s]
 99%|█████████▉| 1534/1545 [13:34<00:05,  2.16it/s]
 99%|█████████▉| 1535/1545 [13:34<00:05,  1.99it/s]
 99%|█████████▉| 1536/1545 [13:35<00:04,  1.90it/s]
 99%|█████████▉| 1537/1545 [13:36<00:04,  1.93it/s]
100%|█████████▉| 1538/1545 [13:36<00:03,  1.81it/s]
100%|█████████▉| 1539/1545 [13:37<00:03,  1.73it/s]
100%|█████████▉| 1540/1545 [13:37<00:02,  1.77it/s]
                                                   
{'loss': 1.0373, 'grad_norm': 9.86623976961809e-18, 'learning_rate': 3.2362459546925574e-08, 'rewards/chosen': -16.71319007873535, 'rewards/rejected': -44.19614028930664, 'rewards/accuracies': 0.800000011920929, 'rewards/margins': 27.482952117919922, 'logps/chosen': -343.7029724121094, 'logps/rejected': -564.3067016601562, 'logits/chosen': -1.7413575649261475, 'logits/rejected': -2.7366671562194824, 'epoch': 1.0}

100%|█████████▉| 1540/1545 [13:37<00:02,  1.77it/s]
100%|█████████▉| 1541/1545 [13:38<00:02,  1.77it/s]
100%|█████████▉| 1542/1545 [13:39<00:01,  1.73it/s]
100%|█████████▉| 1543/1545 [13:39<00:01,  1.72it/s]
100%|█████████▉| 1544/1545 [13:40<00:00,  1.82it/s]
100%|██████████| 1545/1545 [13:40<00:00,  1.79it/s]
                                                   
{'train_runtime': 832.5695, 'train_samples_per_second': 1.856, 'train_steps_per_second': 1.856, 'train_loss': 0.6777341416610262, 'epoch': 1.0}

100%|██████████| 1545/1545 [13:52<00:00,  1.79it/s]
100%|██████████| 1545/1545 [13:52<00:00,  1.86it/s]