[INST] Create an MCQ on the applications of deep learning.\n",
+ "\n",
+ "Here's some context: Examples of Deep Learning:\\nDeep Learning is a type of Machine Learning that uses artificial neural networks with multiple layers to learn and make decisions.\\n....[/INST]\n",
+ "\n",
+ "[question]: Which of the following is an application of deep learning? [option A]: Analyzing financial transactions for fraud detection. [option B]: Predicting future stock prices. [option C]: Recognizing faces in photos. [option D]: All of the above. [correct_answer]: D, [explanation]:Deep learning is used in various applications, including analyzing financial transactions for fraud detection, predicting future stock prices, and recognizing faces in photos.\"}\n",
+ "\n",
+ "```\n",
+ "\n",
+ "We will create a prompt template and a function to apply the template to all the samples in our dataset. Note that we also append a eos token to the end of the sample. This is so that the fine-tuned model will learn to end the sentence at the appropriate time (e.g. end of the instructions) instead of generating tokens indefinitely."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "print(dataset_train)"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "lQhBEMPc0IYs",
+ "outputId": "2cd3f9f5-9fd1-4c7c-ab6a-cd684fb04bcf"
+ },
+ "id": "lQhBEMPc0IYs",
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Dataset({\n",
+ " features: ['expected_output', 'instruction', 'input_content'],\n",
+ " num_rows: 684\n",
+ "})\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "#https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing#scrollTo=NWbzDeSKmakC\n",
+ "#follow llama2 documentation to format the dataset for finetune\n",
+ "\n",
+ "def format_dolly(sample):\n",
+ " instruction = f\"[INST] {sample['instruction']}\"\n",
+ " context = f\"Here's some context: {sample['input_content']}\" if len(sample[\"input_content\"]) > 0 else None\n",
+ " response = f\" [/INST] {sample['expected_output']}\"\n",
+ " # join all the parts together\n",
+ " prompt = \"\".join([i for i in [instruction, context, response] if i is not None])\n",
+ " return prompt\n",
+ "\n",
+ "# template dataset to add prompt to each sample\n",
+ "def template_dataset(sample):\n",
+ " sample[\"text\"] = f\"{format_dolly(sample)}{tokenizer.eos_token}\"\n",
+ " return sample\n",
+ "\n",
+ "# apply prompt template per sample\n",
+ "#dataset = load_dataset(\"databricks/databricks-dolly-15k\", split=\"train\")\n",
+ "\n",
+ "# Shuffle the dataset\n",
+ "dataset_shuffled = dataset_train.shuffle(seed=42)\n",
+ "#dataset_shuffled = dataset_train\n",
+ "\n",
+ "# Select the first 50 rows from the shuffled dataset, comment if you want 15k\n",
+ "#dataset = dataset_shuffled.select(range(50))\n",
+ "\n",
+ "dataset_train = dataset_train.map(template_dataset, remove_columns=list(dataset_shuffled.features))\n",
+ "dataset_train"
+ ],
+ "metadata": {
+ "id": "4gPGgHJGn8yw",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 118,
+ "referenced_widgets": [
+ "3672110eb6dd49a99dd92aa76c5f2ac3",
+ "b19f962c480d43aea1fbaf676510cdbf",
+ "06b1429e860a478d91ea147dea8c6709",
+ "84d773bc0ce44381b3cfd4fa5539eba6",
+ "ccb0b0b4645148439df1d4a596b9656c",
+ "f0336e554d564507b00e4996146e40a0",
+ "bdebc1941c9f418eb0cd72ada0814cb9",
+ "7128ac50f6c24fb599922da873a19477",
+ "566a0d601ca84b9d9cd0575a3bac2319",
+ "5c933e56d8e54796b198d580ca453a56",
+ "526a45b7fdc34b9aab42101b63203d20"
+ ]
+ },
+ "outputId": "7c862fc5-3da8-4265-867e-cd53197645bb"
+ },
+ "execution_count": null,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "Map: 0%| | 0/684 [00:00[INST] Create an MCQ on the applications of deep learningHere's some context: Examples of Deep Learning:\\nDeep Learning is a type of Machine Learning that uses artificial neural networks with multiple layers to learn and make decisions.\\nHere are some examples of Deep Learning:\\n\\nImage and video recognition: Deep learning algorithms are used in image and video recognition systems to classify and analyze visual data. These systems are used in self-driving cars, security systems, and medical imaging.\\nGenerative models: Deep learning algorithms are used in generative models to create new content based on existing data. These systems are used in image and video generation, text generation, and other applications.\\nAutonomous vehicles: Deep learning algorithms are used in self-driving cars and other autonomous vehicles to analyze sensor data and make decisions about speed, direction, and other factors.\\nImage classification: Deep Learning algorithms are used to recognize objects and scenes in images, such as recognizing faces in photos or identifying items in an image for an e-commerce website.\\nSpeech recognition: Deep Learning algorithms are used to transcribe spoken words into text, allowing for voice-controlled interfaces and dictation software.\\nNatural language processing: Deep Learning algorithms are used for tasks such as sentiment analysis, language translation, and text generation.\\nRecommender systems: Deep Learning algorithms are used in recommendation systems to make personalized recommendations based on users' behavior and preferences.\\nFraud detection: Deep Learning algorithms are used in financial transactions to detect patterns of behavior that are indicative of fraud, such as unusual spending patterns or transactions from unfamiliar locations.\\nGame-playing AI: Deep Learning algorithms have been used to develop game-playing AI that can compete at a superhuman level, such as the AlphaGo AI that defeated the world champion in the game of Go.\\nTime series forecasting: Deep Learning algorithms are used to forecast future values in time series data, such as stock prices, energy consumption, and weather patterns. [/INST] [question]: Which of the following is an application of deep learning? [option A]: Analyzing financial transactions for fraud detection. [option B]: Predicting future stock prices. [option C]: Recognizing faces in photos. [option D]: All of the above. [correct_answer]: D, [explanation]:Deep learning is used in various applications, including analyzing financial transactions for fraud detection, predicting future stock prices, and recognizing faces in photos.\"}\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(dataset_train[640])\n",
+ "#print(dataset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "66cc3398-53ac-40a0-9c01-5cb5f02e0bd8",
+ "metadata": {
+ "id": "66cc3398-53ac-40a0-9c01-5cb5f02e0bd8"
+ },
+ "source": [
+ "Similarly we will apply the prompt template to the validation and test splits too."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "69930bd2-f847-433d-8144-c7a2fa732b15",
+ "metadata": {
+ "tags": [],
+ "id": "69930bd2-f847-433d-8144-c7a2fa732b15",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 81,
+ "referenced_widgets": [
+ "579215ad6e744e61bb91893124a9c58a",
+ "84685805ebfa4a1d8d2986bfccae6808",
+ "7b13bdc3c5cb431087deefe11b1b5ebc",
+ "a5dae96c334d4694b707edbd020d5a46",
+ "de07367a319f434cae1dd482757fac5e",
+ "48bcc2af51844e3693bc045ee4544872",
+ "28c9773beb764fdca7d1c529a8315f6c",
+ "2698e9a350014ad0a7886e6751e4f815",
+ "94fdc6ea63b141b2ae83851a8b20c688",
+ "498913909aab4093b7426b95c918973b",
+ "bca29b806ea74e21849679c0af607bba",
+ "0d4f001964244ea699121c10819b4df8",
+ "ebbe52e8c7aa455ba3d38358a4d307d0",
+ "c78c41a66f174b8c98837420520feb04",
+ "e657670203944d238a4971a46883198f",
+ "393d5dffa7f94630bd9136179715464e",
+ "ca8f0c8bcc1e45c6a93bd5a9449a93cc",
+ "207c1ee75acd41a29908e744aefa5af9",
+ "7fa35b3ad21f434c842fc736ea9d4ce5",
+ "fb67a557c456408daad84c10c457eaf7",
+ "590426ba40f94acf98e3172d0930e013",
+ "c8145a501960433cb9c0c6fb3dff49f3"
+ ]
+ },
+ "outputId": "379dd626-3104-44bb-be8d-9929e6eb30ef"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "Map: 0%| | 0/147 [00:00\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cb53e5ba-f34a-46bc-8f6b-0a0414a97bd3",
+ "metadata": {
+ "id": "cb53e5ba-f34a-46bc-8f6b-0a0414a97bd3"
+ },
+ "source": [
+ "The code below help us find the maximum context window of the model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "310776bb-8f44-4199-925a-031d271c53bd",
+ "metadata": {
+ "tags": [],
+ "id": "310776bb-8f44-4199-925a-031d271c53bd",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "6f8439b5-d1f9-4920-932d-82e20c13303e"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Found max context lenth: 4096 in max_position_embeddings\n",
+ "Maximum Context length: 4096\n"
+ ]
+ }
+ ],
+ "source": [
+ "def get_max_context_length(model):\n",
+ "\n",
+ " conf = model.config\n",
+ " max_length = None\n",
+ "\n",
+ " for length_setting in [\"n_positions\", \"max_position_embeddings\", \"seq_length\"]:\n",
+ " max_length = getattr(model.config, length_setting, None)\n",
+ " if max_length:\n",
+ " print(f\"Found max context lenth: {max_length} in {length_setting}\")\n",
+ " break\n",
+ " if not max_length:\n",
+ " max_length = 1024\n",
+ " print(f\"Using default max context length: {max_length}\")\n",
+ "\n",
+ " return max_length\n",
+ "\n",
+ "max_context_length = get_max_context_length(model)\n",
+ "print('Maximum Context length: ', max_context_length)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f75a8059-64c8-41dd-a083-4171dcb4cd7c",
+ "metadata": {
+ "id": "f75a8059-64c8-41dd-a083-4171dcb4cd7c"
+ },
+ "source": [
+ "The following functions concatenate a batch of samples, and then divide the concatenated sample into chunks of context size. Also we also need to create 'labels' in the input dataset, which tells the model what is the token to be predicted. Shifting the inputs and labels to align them happens inside the model, so our labels are just the exact copy of the input_ids.\n",
+ "\n",
+ "In the code below, we use a context_length of 512 instad of the maximum 4096, as we have limited gpu memory and using a larger context length will result in Out of Memory error even with batch size of 1."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bd6d2b52-fdec-4c36-9be4-c9922dcb62f3",
+ "metadata": {
+ "tags": [],
+ "id": "bd6d2b52-fdec-4c36-9be4-c9922dcb62f3"
+ },
+ "outputs": [],
+ "source": [
+ "context_length = 512\n",
+ "# context_length = max_context_length\n",
+ "\n",
+ "def group_texts(examples):\n",
+ "\n",
+ " # Concatenate all texts.\n",
+ " concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n",
+ " total_length = len(concatenated_examples[list(examples.keys())[0]])\n",
+ " # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n",
+ " # customize this part to your needs.\n",
+ " if total_length >= context_length:\n",
+ " total_length = (total_length // context_length) * context_length\n",
+ " # Split by chunks of context length.\n",
+ " result = {\n",
+ " k: [t[i : i + context_length] for i in range(0, total_length, context_length)]\n",
+ " for k, t in concatenated_examples.items()\n",
+ " }\n",
+ " result[\"labels\"] = result[\"input_ids\"].copy()\n",
+ " return result\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "de68071b-2cc9-4931-abbc-cb495c83f538",
+ "metadata": {
+ "tags": [],
+ "id": "de68071b-2cc9-4931-abbc-cb495c83f538",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 113,
+ "referenced_widgets": [
+ "50bda5240cd543e09a716cc7c988d7b0",
+ "740ba74944a14b498e2618337e82940e",
+ "59876c6fa9b846ccb702dd475a9eec61",
+ "0a869517810d408695e96b49be84f130",
+ "e9491a6eeca04df9974f4a1a1c7757c5",
+ "667209adc7f34aedafd9d8d758294d87",
+ "1d9d5cc6c71a42918810ba6432fa58c1",
+ "6e2961091ed24ab592b2f922dc2377cc",
+ "9238d40a556d4a3b85b1523808749525",
+ "e3d8e2c63fb84e67b3e5a339747741d2",
+ "2eb49ff221904895b5af04bf703d3e67",
+ "cca91bc1cb8f407eae07c27551596c2b",
+ "de1b9004e5b64923849833f7e106841e",
+ "4183a35153944c6bab5893b24e829243",
+ "9957f75cd868441d81dbe377a5a8eecd",
+ "1f59f77e260248b6a941c1b15c99cb68",
+ "b4455050bb4a41e6a6ca086d95c0a0b2",
+ "daa129435c0b4ad7af27e58add9073fd",
+ "f2af85f632174017826028cac3f058a7",
+ "4bba9920ca264bc09600af041f5c2869",
+ "6300bb75763f4379bd13b32d73d72197",
+ "e23eac9adf7f44a6a1b38b4b4fc73c7e",
+ "d77982eb21094cc38dacbe216cf874dd",
+ "e9ced280518047cc881226d555ed09d7",
+ "3f2791500bc34577918b595172a0236b",
+ "1bec54c87d74442a80f24752da12e80f",
+ "9b67df6eafcb44a596b229f6439def72",
+ "ca053a27667e40c48d597b0eb98b0cda",
+ "a94a7f3e0dd844af9f31699e29d7ec32",
+ "dbfe8c0c14944f268512d14592a68a50",
+ "384d926f77b34faa8dc45b5079f70cd6",
+ "525c98a8fe9a4fb3bf85306e93d14f62",
+ "1056ace3f67748dd8802da5c8dd8c550"
+ ]
+ },
+ "outputId": "5db2687c-9da6-4d05-b4a2-e6d59a46abf6"
+ },
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "Map (num_proc=4): 0%| | 0/684 [00:00"
+ ],
+ "text/html": [
+ "\n",
+ " \n",
+ " \n",
+ "
\n",
+ " [200/200 59:49, Epoch 3/4]\n",
+ "
\n",
+ " \n",
+ " \n",
+ " | Step | \n",
+ " Training Loss | \n",
+ " Validation Loss | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | 10 | \n",
+ " 1.681300 | \n",
+ " 1.631856 | \n",
+ "
\n",
+ " \n",
+ " | 20 | \n",
+ " 1.583000 | \n",
+ " 1.498535 | \n",
+ "
\n",
+ " \n",
+ " | 30 | \n",
+ " 1.413800 | \n",
+ " 1.393412 | \n",
+ "
\n",
+ " \n",
+ " | 40 | \n",
+ " 1.360300 | \n",
+ " 1.308431 | \n",
+ "
\n",
+ " \n",
+ " | 50 | \n",
+ " 1.271100 | \n",
+ " 1.237813 | \n",
+ "
\n",
+ " \n",
+ " | 60 | \n",
+ " 1.225500 | \n",
+ " 1.179109 | \n",
+ "
\n",
+ " \n",
+ " | 70 | \n",
+ " 1.168600 | \n",
+ " 1.139814 | \n",
+ "
\n",
+ " \n",
+ " | 80 | \n",
+ " 1.082900 | \n",
+ " 1.112842 | \n",
+ "
\n",
+ " \n",
+ " | 90 | \n",
+ " 1.114000 | \n",
+ " 1.087292 | \n",
+ "
\n",
+ " \n",
+ " | 100 | \n",
+ " 1.077200 | \n",
+ " 1.062716 | \n",
+ "
\n",
+ " \n",
+ " | 110 | \n",
+ " 1.020000 | \n",
+ " 1.039254 | \n",
+ "
\n",
+ " \n",
+ " | 120 | \n",
+ " 0.994900 | \n",
+ " 1.012544 | \n",
+ "
\n",
+ " \n",
+ " | 130 | \n",
+ " 0.990900 | \n",
+ " 0.976476 | \n",
+ "
\n",
+ " \n",
+ " | 140 | \n",
+ " 0.910700 | \n",
+ " 0.923615 | \n",
+ "
\n",
+ " \n",
+ " | 150 | \n",
+ " 0.871000 | \n",
+ " 0.896943 | \n",
+ "
\n",
+ " \n",
+ " | 160 | \n",
+ " 0.897800 | \n",
+ " 0.883769 | \n",
+ "
\n",
+ " \n",
+ " | 170 | \n",
+ " 0.869700 | \n",
+ " 0.875816 | \n",
+ "
\n",
+ " \n",
+ " | 180 | \n",
+ " 0.867600 | \n",
+ " 0.870564 | \n",
+ "
\n",
+ " \n",
+ " | 190 | \n",
+ " 0.797800 | \n",
+ " 0.867839 | \n",
+ "
\n",
+ " \n",
+ " | 200 | \n",
+ " 0.804900 | \n",
+ " 0.867121 | \n",
+ "
\n",
+ " \n",
+ "
"
+ ]
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+ "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
+ " warnings.warn(\n",
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n"
+ ]
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "TrainOutput(global_step=200, training_loss=1.100144896507263, metrics={'train_runtime': 3605.5794, 'train_samples_per_second': 0.222, 'train_steps_per_second': 0.055, 'total_flos': 1.6248515592192e+16, 'train_loss': 1.100144896507263, 'epoch': 3.25})"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 33
+ }
+ ],
+ "source": [
+ "# Start training\n",
+ "trainer.train()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "93524441-36a1-4f7e-8307-b855e68c5c14",
+ "metadata": {
+ "tags": [],
+ "id": "93524441-36a1-4f7e-8307-b855e68c5c14",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 178
+ },
+ "outputId": "2bba0ba1-3427-4b6e-8cf9-258363851fe5"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": [
+ "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
+ " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n"
+ ]
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ ""
+ ],
+ "text/html": [
+ "\n",
+ " \n",
+ " \n",
+ "
\n",
+ " [25/25 00:27]\n",
+ "
Copy a token from your Hugging Face\ntokens page and paste it below.
Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. "
+ }
+ },
+ "e463243dece7453da77c8775a4fe5f94": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "PasswordModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "PasswordModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "PasswordView",
+ "continuous_update": true,
+ "description": "Token:",
+ "description_tooltip": null,
+ "disabled": false,
+ "layout": "IPY_MODEL_de3d1e34487d4763a2b19c8f7567c8f3",
+ "placeholder": "",
+ "style": "IPY_MODEL_07bc154a336b402f9156c30cba0ed696",
+ "value": ""
+ }
+ },
+ "44344f5882fd4e3aadc05cc18b15d571": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "CheckboxModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "CheckboxModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "CheckboxView",
+ "description": "Add token as git credential?",
+ "description_tooltip": null,
+ "disabled": false,
+ "indent": true,
+ "layout": "IPY_MODEL_ac978db16d5f442d8930984fa4498717",
+ "style": "IPY_MODEL_5e0f629e3e094748b2f03675e4370989",
+ "value": true
+ }
+ },
+ "eb26fc1a3e6047078528f3f92ccd27b9": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "ButtonModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "ButtonModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "ButtonView",
+ "button_style": "",
+ "description": "Login",
+ "disabled": false,
+ "icon": "",
+ "layout": "IPY_MODEL_498c2dac23ff4355b3475328c7d77deb",
+ "style": "IPY_MODEL_96f915a83bb843b5913508c7cf7debe4",
+ "tooltip": ""
+ }
+ },
+ "53927a52e58544b6bdbb71382a3f0d45": {
+ "model_module": "@jupyter-widgets/controls",
+ "model_name": "HTMLModel",
+ "model_module_version": "1.5.0",
+ "state": {
+ "_dom_classes": [],
+ "_model_module": "@jupyter-widgets/controls",
+ "_model_module_version": "1.5.0",
+ "_model_name": "HTMLModel",
+ "_view_count": null,
+ "_view_module": "@jupyter-widgets/controls",
+ "_view_module_version": "1.5.0",
+ "_view_name": "HTMLView",
+ "description": "",
+ "description_tooltip": null,
+ "layout": "IPY_MODEL_315c11cdbd3b4fcf93881deae18b8a4b",
+ "placeholder": "",
+ "style": "IPY_MODEL_8e1c860e75bd42a2ae54047ba436ea01",
+ "value": "\nPro Tip: If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks.