\n",
- " datasets.arrow_dataset.Dataset
def __init__(arrow_table: Table, info: Optional[DatasetInfo]=None, split: Optional[NamedSplit]=None, indices_table: Optional[Table]=None, fingerprint: Optional[str]=None)
/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.pyA Dataset backed by an Arrow table.
\n",
- " \n",
- " "
- ]
- },
- "metadata": {},
- "execution_count": 12
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "print(dataset_train)\n",
- "print(dataset_test)\n",
- "print(dataset_val)"
- ],
- "metadata": {
- "id": "juh2RGTHjApA",
- "outputId": "38815e79-7b61-4476-8974-cbb2e39ccbfe",
- "colab": {
- "base_uri": "https://localhost:8080/"
- }
- },
- "id": "juh2RGTHjApA",
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Dataset({\n",
- " features: ['expected_output', 'instruction', 'input_content'],\n",
- " num_rows: 684\n",
- "})\n",
- "Dataset({\n",
- " features: ['expected_output', 'instruction', 'input_content'],\n",
- " num_rows: 147\n",
- "})\n",
- "Dataset({\n",
- " features: ['expected_output', 'instruction', 'input_content'],\n",
- " num_rows: 147\n",
- "})\n"
- ]
- }
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f5b7da07-9192-4718-b10e-c47fa5634c02",
- "metadata": {
- "id": "f5b7da07-9192-4718-b10e-c47fa5634c02"
- },
- "source": [
- "Let's taka a look at one of the samples"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f1ce8a86-3bc2-44d6-9cf1-3dcda2085ede",
- "metadata": {
- "tags": [],
- "id": "f1ce8a86-3bc2-44d6-9cf1-3dcda2085ede",
- "outputId": "726ded67-4ddd-4601-f17e-b271b48b336b",
- "colab": {
- "base_uri": "https://localhost:8080/"
- }
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "{'expected_output': '[question]: Which of the following is an application of machine learning in image recognition? [option A]: Analyzing financial transactions for fraud detection. [option B]: Transcribing spoken words into text. [option C]: Classifying images based on their contents. [option D]: Predicting when machinery is likely to fail. [correct_answer]: C, [explanation]:Machine learning algorithms are used in image recognition systems to classify images based on their contents.',\n",
- " 'instruction': 'Create an MCQ on the application of machine learning in image recognition',\n",
- " 'input_content': ''}"
- ]
- },
- "metadata": {},
- "execution_count": 16
- }
- ],
- "source": [
- "dataset['train'][600]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e8358e2e-e29d-4d5c-89f9-f78959ef5527",
- "metadata": {
- "id": "e8358e2e-e29d-4d5c-89f9-f78959ef5527"
- },
- "source": [
- "## Test the Model with Zero Shot Inferencing\n",
- "\n",
- "Let's test the model with zero shot inferencing (i.e. ask it to summarize without giving any example. You can see that the model struggles to summarize the dialogue compared to the baseline summary, and it is just repeating the conversation."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "31def3d5-e863-4b34-8c2e-408a0cb8f0b4",
- "metadata": {
- "tags": [],
- "id": "31def3d5-e863-4b34-8c2e-408a0cb8f0b4",
- "outputId": "2d4059e9-4efd-4ac2-82e9-6f35dbaeb146",
- "colab": {
- "base_uri": "https://localhost:8080/"
- }
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "\n",
- "Create an Multiple choice question:\n",
- "Artificial neural networks are built on the principles of the structure and operation of human neurons. \n",
- "It is also known as neural networks or neural nets. An artificial neural network’s input layer, which is the first layer, receives input from external sources and passes it on to the hidden layer, which is the second layer. \n",
- "Each neuron in the hidden layer gets information from the neurons in the previous layer, computes the weighted total, and then transfers it to the neurons in the next layer. \n",
- "These connections are weighted, which means that the impacts of the inputs from the preceding layer are more or less optimized by giving each input a distinct weight. \n",
- "These weights are then adjusted during the training process to enhance the performance of the model. \n",
- "Artificial neurons, also known as units, are found in artificial neural networks. \n",
- "The whole Artificial Neural Network is composed of these artificial neurons, which are arranged in a series of layers. \n",
- "The complexities of neural networks will depend on the complexities of the underlying patterns in the dataset whether a layer has a dozen units or millions of units. \n",
- "Commonly, Artificial Neural Network has an input layer, an output layer as well as hidden layers. \n",
- "The input layer receives data from the outside world which the neural network needs to analyze or learn about. \n",
- "In a fully connected artificial neural network, there is an input layer and one or more hidden layers connected one after the other. \n",
- "Each neuron receives input from the previous layer neurons or the input layer. The output of one neuron becomes the input to other neurons in the next layer of the network, and this process continues until the final layer produces the output of the network. \n",
- "Then, after passing through one or more hidden layers, this data is transformed into valuable data for the output layer. Finally, the output layer provides an output in the form of an artificial neural network’s response to the data that comes in.\n",
- "\n",
- "---\n",
- "question:\n",
- "options A:\n",
- "options B:\n",
- "options C:\n",
- "options D:\n",
- "correct_answer:\n",
- "explanation: \n",
- "Please select the correct answer from the options given below.\n",
- "\n",
- "A) Artificial neural networks are built on the principles of the structure and operation of human neurons.\n",
- "B) An artificial neural network’s input layer, which is the first layer, receives input from external sources and passes it on to the hidden layer, which is the second layer.\n",
- "C) Each neuron in the hidden layer gets information from the neurons in the previous layer, computes the weighted total, and then transfers it to the neurons in the next layer.\n",
- "D) Artificial neurons, also known as units, are found in artificial neural networks.\n",
- "\n",
- "Please select the correct answer from the options given below.\n"
- ]
- }
- ],
- "source": [
- "eval_prompt = \"\"\"\n",
- "Create an Multiple choice question:\n",
- "Artificial neural networks are built on the principles of the structure and operation of human neurons.\n",
- "It is also known as neural networks or neural nets. An artificial neural network\\u2019s input layer, which is the first layer, receives input from external sources and passes it on to the hidden layer, which is the second layer.\n",
- "Each neuron in the hidden layer gets information from the neurons in the previous layer, computes the weighted total, and then transfers it to the neurons in the next layer.\n",
- "These connections are weighted, which means that the impacts of the inputs from the preceding layer are more or less optimized by giving each input a distinct weight.\n",
- "These weights are then adjusted during the training process to enhance the performance of the model.\n",
- "Artificial neurons, also known as units, are found in artificial neural networks.\n",
- "The whole Artificial Neural Network is composed of these artificial neurons, which are arranged in a series of layers.\n",
- "The complexities of neural networks will depend on the complexities of the underlying patterns in the dataset whether a layer has a dozen units or millions of units.\n",
- "Commonly, Artificial Neural Network has an input layer, an output layer as well as hidden layers.\n",
- "The input layer receives data from the outside world which the neural network needs to analyze or learn about.\n",
- "In a fully connected artificial neural network, there is an input layer and one or more hidden layers connected one after the other.\n",
- "Each neuron receives input from the previous layer neurons or the input layer. The output of one neuron becomes the input to other neurons in the next layer of the network, and this process continues until the final layer produces the output of the network.\n",
- "Then, after passing through one or more hidden layers, this data is transformed into valuable data for the output layer. Finally, the output layer provides an output in the form of an artificial neural network\\u2019s response to the data that comes in.\n",
- "\n",
- "---\n",
- "question:\n",
- "options A:\n",
- "options B:\n",
- "options C:\n",
- "options D:\n",
- "correct_answer:\n",
- "explanation:\n",
- "\"\"\"\n",
- "\n",
- "model_input = tokenizer(eval_prompt, return_tensors=\"pt\").to(\"cuda\")\n",
- "\n",
- "model.eval()\n",
- "with torch.no_grad(): # no gradient update\n",
- " print(tokenizer.decode(model.generate(**model_input, max_new_tokens=200)[0], skip_special_tokens=True))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cf285722-2ac5-4e60-8cc6-06c048e6d723",
- "metadata": {
- "id": "cf285722-2ac5-4e60-8cc6-06c048e6d723"
- },
- "source": [
- "## Creating instruction dataset\n",
- "\n",
- "We will now prepare our dataset to fine-tune our base model (instruction fine-tuning)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e4012814-8355-4d75-8d0e-e1efbd2ba8de",
- "metadata": {
- "tags": [],
- "id": "e4012814-8355-4d75-8d0e-e1efbd2ba8de"
- },
- "source": [
- "### Instruction prompt\n",
- "\n",
- "We need to convert the insturctions+ input and expected output (prompt-response) pairs into explicit instructions for the LLM such as follows:\n",
- "\n",
- "```\n",
- "{'text': \"[INST] Create an MCQ on the applications of deep learning\n",
- "\n",
- "Here's some context: Examples of Deep Learning:\\nDeep Learning is a type of Machine Learning that uses artificial neural networks with multiple layers to learn and make decisions.\\n....[/INST]\n",
- "\n",
- "[question]: Which of the following is an application of deep learning? [option A]: Analyzing financial transactions for fraud detection. [option B]: Predicting future stock prices. [option C]: Recognizing faces in photos. [option D]: All of the above. [correct_answer]: D, [explanation]:Deep learning is used in various applications, including analyzing financial transactions for fraud detection, predicting future stock prices, and recognizing faces in photos.\"}\n",
- "\n",
- "```\n",
- "\n",
- "We will create a prompt template and a function to apply the template to all the samples in our dataset. Note that we also append a eos token to the end of the sample. This is so that the fine-tuned model will learn to end the sentence at the appropriate time (e.g. end of the instructions) instead of generating tokens indefinitely."
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "print(dataset_train)"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "lQhBEMPc0IYs",
- "outputId": "f1d3f21e-c365-47c3-a32d-d07f7cb46bbc"
- },
- "id": "lQhBEMPc0IYs",
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Dataset({\n",
- " features: ['expected_output', 'instruction', 'input_content'],\n",
- " num_rows: 684\n",
- "})\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "#https://colab.research.google.com/drive/134o_cXcMe_lsvl15ZE_4Y75Kstepsntu?usp=sharing#scrollTo=NWbzDeSKmakC\n",
- "#follow llama2 documentation to format the dataset for finetune\n",
- "\n",
- "def format_dolly(sample):\n",
- " instruction = f\"[INST] {sample['instruction']}\"\n",
- " context = f\"Here's some context: {sample['input_content']}\" if len(sample[\"input_content\"]) > 0 else None\n",
- " response = f\" [/INST] {sample['expected_output']}\"\n",
- " # join all the parts together\n",
- " prompt = \"\".join([i for i in [instruction, context, response] if i is not None])\n",
- " return prompt\n",
- "\n",
- "# template dataset to add prompt to each sample\n",
- "def template_dataset(sample):\n",
- " sample[\"text\"] = f\"{format_dolly(sample)}{tokenizer.eos_token}\"\n",
- " return sample\n",
- "\n",
- "# apply prompt template per sample\n",
- "#dataset = load_dataset(\"databricks/databricks-dolly-15k\", split=\"train\")\n",
- "\n",
- "# Shuffle the dataset\n",
- "dataset_shuffled = dataset_train.shuffle(seed=42)\n",
- "#dataset_shuffled = dataset_train\n",
- "\n",
- "# Select the first 50 rows from the shuffled dataset, comment if you want 15k\n",
- "#dataset = dataset_shuffled.select(range(50))\n",
- "\n",
- "dataset_train = dataset_train.map(template_dataset, remove_columns=list(dataset_shuffled.features))\n",
- "dataset_train"
- ],
- "metadata": {
- "id": "4gPGgHJGn8yw",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 118,
- "referenced_widgets": [
- "c2f9f28cf2914570ac8446a48026a3bf",
- "4dcde00cfe604b9e9109907b046bb18a",
- "055478fbd9d442b28d984b153ad0e5ef",
- "0f472edf561d478ba552a70669c9f3ea",
- "f147916fb3f7465ebf856a198e0c9fab",
- "a4666699cf28473fb4bae8ecb67cbf24",
- "660e7e7d609a4a4b9004a689cbda87dc",
- "daee76f875bf462990cd14c5cba818f9",
- "ab5269cbbae6443dbd2cbb1444a7b56c",
- "ad134097c8c34244bc155d94592e7fab",
- "9762c56a82504232a4e2fb95f16e3f2a"
- ]
- },
- "outputId": "e83d7a2d-1496-4cf1-c915-362bab9020dd"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "Map: 0%| | 0/684 [00:00[INST] Summarize the architecture of artificial neural networksHere's some context: Artificial neural networks have input layers, hidden layers, and output layers. Each layer is composed of artificial neurons or units. In a fully connected network, each neuron receives input from the previous layer or the input layer. The output of one neuron becomes the input for other neurons in the next layer. The final layer produces the network's output. [/INST] Artificial neural networks consist of input, hidden, and output layers. Each layer contains artificial neurons or units. Neurons in a layer receive input from the previous layer or the input layer. The output of one neuron becomes the input for other neurons in the next layer. The final layer produces the network's output.\"}\n"
- ]
- }
- ],
- "source": [
- "print(dataset_train[110])\n",
- "#print(dataset)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "66cc3398-53ac-40a0-9c01-5cb5f02e0bd8",
- "metadata": {
- "id": "66cc3398-53ac-40a0-9c01-5cb5f02e0bd8"
- },
- "source": [
- "Similarly we will apply the prompt template to the validation and test splits too."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "69930bd2-f847-433d-8144-c7a2fa732b15",
- "metadata": {
- "tags": [],
- "id": "69930bd2-f847-433d-8144-c7a2fa732b15",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 81,
- "referenced_widgets": [
- "6eb64758cd9f4921b69655868c428a0c",
- "b1129d71adf643e8b75089e43a29230e",
- "7eecd26c38204a6ab360c456a860f69c",
- "ceb3aeafa97f478ca7ba56cc5b90990b",
- "7abdc553306c4ce59a9d9642d307d7c1",
- "7d9335a47bfb43a99bb13311d4e6088d",
- "c4ba4d02009249b382ba2aa528122ab6",
- "1dd54e75068b4038af0d9fe75ceb2c68",
- "f7524f28fdaf4bc993dad499312165e4",
- "6e3df06c2b3649e69933b740c8edf7e9",
- "820dd3f57b4e457b95da525e00400d22",
- "53f5043b8e0b4842bba04f9d0d2f34b9",
- "38d06ba36cc649d093117b9863178f2d",
- "aeb29a2b58c04632a4b016e470aed464",
- "0245b6685d7645b29bad2288899dab52",
- "bfb2b962a6404adb985f033576334d89",
- "832335249b154b939082e8fe78417dc1",
- "64a44b5e587a4784a87f824858a775df",
- "fac8cab862614352a01cae3b25295fb3",
- "12d2107b4c5a40b591bd8dbb2912ab30",
- "2f4e81a69d344f3691045128fbe03fea",
- "ff8d6446790f406995c73120cd6c5b85"
- ]
- },
- "outputId": "3867b72a-1f80-4760-892f-0c0354b7f429"
- },
- "outputs": [
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "Map: 0%| | 0/147 [00:00\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cb53e5ba-f34a-46bc-8f6b-0a0414a97bd3",
- "metadata": {
- "id": "cb53e5ba-f34a-46bc-8f6b-0a0414a97bd3"
- },
- "source": [
- "The code below help us find the maximum context window of the model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "310776bb-8f44-4199-925a-031d271c53bd",
- "metadata": {
- "tags": [],
- "id": "310776bb-8f44-4199-925a-031d271c53bd",
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "outputId": "ff6600f0-f152-418b-aca5-64bbf0aedada"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Found max context lenth: 4096 in max_position_embeddings\n",
- "Maximum Context length: 4096\n"
- ]
- }
- ],
- "source": [
- "def get_max_context_length(model):\n",
- "\n",
- " conf = model.config\n",
- " max_length = None\n",
- "\n",
- " for length_setting in [\"n_positions\", \"max_position_embeddings\", \"seq_length\"]:\n",
- " max_length = getattr(model.config, length_setting, None)\n",
- " if max_length:\n",
- " print(f\"Found max context lenth: {max_length} in {length_setting}\")\n",
- " break\n",
- " if not max_length:\n",
- " max_length = 1024\n",
- " print(f\"Using default max context length: {max_length}\")\n",
- "\n",
- " return max_length\n",
- "\n",
- "max_context_length = get_max_context_length(model)\n",
- "print('Maximum Context length: ', max_context_length)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f75a8059-64c8-41dd-a083-4171dcb4cd7c",
- "metadata": {
- "id": "f75a8059-64c8-41dd-a083-4171dcb4cd7c"
- },
- "source": [
- "The following functions concatenate a batch of samples, and then divide the concatenated sample into chunks of context size. Also we also need to create 'labels' in the input dataset, which tells the model what is the token to be predicted. Shifting the inputs and labels to align them happens inside the model, so our labels are just the exact copy of the input_ids.\n",
- "\n",
- "In the code below, we use a context_length of 512 instad of the maximum 4096, as we have limited gpu memory and using a larger context length will result in Out of Memory error even with batch size of 1."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "bd6d2b52-fdec-4c36-9be4-c9922dcb62f3",
- "metadata": {
- "tags": [],
- "id": "bd6d2b52-fdec-4c36-9be4-c9922dcb62f3"
- },
- "outputs": [],
- "source": [
- "context_length = 512\n",
- "# context_length = max_context_length\n",
- "\n",
- "def group_texts(examples):\n",
- "\n",
- " # Concatenate all texts.\n",
- " concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\n",
- " total_length = len(concatenated_examples[list(examples.keys())[0]])\n",
- " # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can\n",
- " # customize this part to your needs.\n",
- " if total_length >= context_length:\n",
- " total_length = (total_length // context_length) * context_length\n",
- " # Split by chunks of context length.\n",
- " result = {\n",
- " k: [t[i : i + context_length] for i in range(0, total_length, context_length)]\n",
- " for k, t in concatenated_examples.items()\n",
- " }\n",
- " result[\"labels\"] = result[\"input_ids\"].copy()\n",
- " return result\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "de68071b-2cc9-4931-abbc-cb495c83f538",
- "metadata": {
- "tags": [],
- "id": "de68071b-2cc9-4931-abbc-cb495c83f538",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 113,
- "referenced_widgets": [
- "c10a1824b4964aadb2cba95d85c64cde",
- "3acd9492f1ab4b0a89bbcc587ff6ebfa",
- "1422dd0d7da342a19786a8e60f00a463",
- "90d29862ac6a4686bb252daded081bde",
- "3be5f0ddab7c475c979e7826b82e3c9a",
- "41d493b1a90c421fa4db6e1585067bbd",
- "3b4a622702b748e293cd6db9381acad6",
- "627cb0ab7acc4278988e4b8a6509d1a2",
- "9c354a9dc9b742be90a1deb9579f1be2",
- "15a848cb1a2c48b5ac3b8f1376d15ed7",
- "7ddf72e5b1154db5b4555191c5346bbb",
- "71ed459b656f4dc3a5749213894e306c",
- "c405d7e18f374a3b94e9dbbd650c311f",
- "8d1818d9f1b94d1abc8a7b699beac5c4",
- "4d15938847a74482ae2c6fa1fc38822f",
- "9852573b36bd41f789cecb5906889131",
- "f599fc9873c848cd892878bed113fe4b",
- "d67f8a418fdb40c4ac6fc78e3f24f408",
- "ec25383094b4470f812d1f2feaa0430a",
- "53266e2619f34c439972f702f40a14fe",
- "95d5043ad4bf40f68ad3583715213c8a",
- "3023427cf5ff412ba27dacb274f2f3a8",
- "7d6f59c3384248ba8bcd17bd507813dd",
- "0defae97c2784f67865153860c57d2d0",
- "77a08c827ec547e1ab12571c38cb40a0",
- "7f812718c13f40f1ab59249eb5053655",
- "398f4c2e4fe34a5cb06088dc03c72e0d",
- "401f4e5fa90e4f128b8bc7afec034200",
- "ef7b2b94875b42819079c32f57e0f04b",
- "716f95a5bd4c46ff8e8ab0d2f343651c",
- "aca2b090d0cf42b49a1e52cc5e52ee94",
- "6e92f1e49d574a668177a61bab778415",
- "f9f640dc53c54086ad9f2bb28a7170c0"
- ]
- },
- "outputId": "29fa1150-d262-4244-f811-3aeca933bd85"
- },
- "outputs": [
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "Map (num_proc=4): 0%| | 0/684 [00:00\n",
- " \n",
- " \n",
- " [ 80/200 22:48 < 35:05, 0.06 it/s, Epoch 1.28/4]\n",
- " \n",
- " \n",
- " \n",
- " \n",
- " | Step | \n",
- " Training Loss | \n",
- " Validation Loss | \n",
- "
\n",
- " \n",
- " \n",
- " \n",
- " | 10 | \n",
- " 2.041100 | \n",
- " 1.894824 | \n",
- "
\n",
- " \n",
- " | 20 | \n",
- " 1.784500 | \n",
- " 1.614028 | \n",
- "
\n",
- " \n",
- " | 30 | \n",
- " 1.487900 | \n",
- " 1.455886 | \n",
- "
\n",
- " \n",
- " | 40 | \n",
- " 1.400600 | \n",
- " 1.362805 | \n",
- "
\n",
- " \n",
- " | 50 | \n",
- " 1.339200 | \n",
- " 1.303091 | \n",
- "
\n",
- " \n",
- " | 60 | \n",
- " 1.300000 | \n",
- " 1.256285 | \n",
- "
\n",
- " \n",
- " | 70 | \n",
- " 1.255700 | \n",
- " 1.216590 | \n",
- "
\n",
- " \n",
- "
"
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "metadata": {
- "tags": null
- },
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n"
- ]
- },
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- ""
- ],
- "text/html": [
- "\n",
- " \n",
- " \n",
- "
\n",
- " [200/200 58:59, Epoch 3/4]\n",
- "
\n",
- " \n",
- " \n",
- " | Step | \n",
- " Training Loss | \n",
- " Validation Loss | \n",
- "
\n",
- " \n",
- " \n",
- " \n",
- " | 10 | \n",
- " 2.041100 | \n",
- " 1.894824 | \n",
- "
\n",
- " \n",
- " | 20 | \n",
- " 1.784500 | \n",
- " 1.614028 | \n",
- "
\n",
- " \n",
- " | 30 | \n",
- " 1.487900 | \n",
- " 1.455886 | \n",
- "
\n",
- " \n",
- " | 40 | \n",
- " 1.400600 | \n",
- " 1.362805 | \n",
- "
\n",
- " \n",
- " | 50 | \n",
- " 1.339200 | \n",
- " 1.303091 | \n",
- "
\n",
- " \n",
- " | 60 | \n",
- " 1.300000 | \n",
- " 1.256285 | \n",
- "
\n",
- " \n",
- " | 70 | \n",
- " 1.255700 | \n",
- " 1.216590 | \n",
- "
\n",
- " \n",
- " | 80 | \n",
- " 1.150800 | \n",
- " 1.181747 | \n",
- "
\n",
- " \n",
- " | 90 | \n",
- " 1.175700 | \n",
- " 1.151433 | \n",
- "
\n",
- " \n",
- " | 100 | \n",
- " 1.148600 | \n",
- " 1.126270 | \n",
- "
\n",
- " \n",
- " | 110 | \n",
- " 1.083100 | \n",
- " 1.106496 | \n",
- "
\n",
- " \n",
- " | 120 | \n",
- " 1.075300 | \n",
- " 1.090339 | \n",
- "
\n",
- " \n",
- " | 130 | \n",
- " 1.081900 | \n",
- " 1.076817 | \n",
- "
\n",
- " \n",
- " | 140 | \n",
- " 1.029900 | \n",
- " 1.066747 | \n",
- "
\n",
- " \n",
- " | 150 | \n",
- " 1.034400 | \n",
- " 1.057581 | \n",
- "
\n",
- " \n",
- " | 160 | \n",
- " 1.055500 | \n",
- " 1.050728 | \n",
- "
\n",
- " \n",
- " | 170 | \n",
- " 1.042700 | \n",
- " 1.044938 | \n",
- "
\n",
- " \n",
- " | 180 | \n",
- " 1.039200 | \n",
- " 1.041671 | \n",
- "
\n",
- " \n",
- " | 190 | \n",
- " 0.981000 | \n",
- " 1.039009 | \n",
- "
\n",
- " \n",
- " | 200 | \n",
- " 0.978100 | \n",
- " 1.038632 | \n",
- "
\n",
- " \n",
- "
"
- ]
- },
- "metadata": {}
- },
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
- "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
- " warnings.warn(\n",
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n"
- ]
- },
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "TrainOutput(global_step=200, training_loss=1.2242571496963501, metrics={'train_runtime': 3556.5472, 'train_samples_per_second': 0.225, 'train_steps_per_second': 0.056, 'total_flos': 1.6248515592192e+16, 'train_loss': 1.2242571496963501, 'epoch': 3.25})"
- ]
- },
- "metadata": {},
- "execution_count": 33
- }
- ],
- "source": [
- "# Start training\n",
- "\n",
- "trainer.train()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "93524441-36a1-4f7e-8307-b855e68c5c14",
- "metadata": {
- "tags": [],
- "id": "93524441-36a1-4f7e-8307-b855e68c5c14",
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 178
- },
- "outputId": "b6424c6a-3ff9-47e9-9a7c-f0137908a733"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py:322: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
- " warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n"
- ]
- },
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- ""
- ],
- "text/html": [
- "\n",
- " \n",
- " \n",
- "
\n",
- " [25/25 00:29]\n",
- "
Copy a token from your Hugging Face\ntokens page and paste it below.
Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. "
- }
- },
- "012d0b97008247b9a03142cf26d0e381": {
- "model_module": "@jupyter-widgets/controls",
- "model_name": "PasswordModel",
- "model_module_version": "1.5.0",
- "state": {
- "_dom_classes": [],
- "_model_module": "@jupyter-widgets/controls",
- "_model_module_version": "1.5.0",
- "_model_name": "PasswordModel",
- "_view_count": null,
- "_view_module": "@jupyter-widgets/controls",
- "_view_module_version": "1.5.0",
- "_view_name": "PasswordView",
- "continuous_update": true,
- "description": "Token:",
- "description_tooltip": null,
- "disabled": false,
- "layout": "IPY_MODEL_5446b40d53804682948236d4ba98cdd6",
- "placeholder": "",
- "style": "IPY_MODEL_21ea43357d7e4b2d8b2a3f12d199a3ce",
- "value": ""
- }
- },
- "7250673145e042cb8c952fdf3477f5a6": {
- "model_module": "@jupyter-widgets/controls",
- "model_name": "CheckboxModel",
- "model_module_version": "1.5.0",
- "state": {
- "_dom_classes": [],
- "_model_module": "@jupyter-widgets/controls",
- "_model_module_version": "1.5.0",
- "_model_name": "CheckboxModel",
- "_view_count": null,
- "_view_module": "@jupyter-widgets/controls",
- "_view_module_version": "1.5.0",
- "_view_name": "CheckboxView",
- "description": "Add token as git credential?",
- "description_tooltip": null,
- "disabled": false,
- "indent": true,
- "layout": "IPY_MODEL_eb24b1c2c039416a8b63910c9d254147",
- "style": "IPY_MODEL_0e3de9ad4e1a4affbe73c1ee046c6105",
- "value": true
- }
- },
- "8688ae1d1fd544b39cd6b043bbf9e957": {
- "model_module": "@jupyter-widgets/controls",
- "model_name": "ButtonModel",
- "model_module_version": "1.5.0",
- "state": {
- "_dom_classes": [],
- "_model_module": "@jupyter-widgets/controls",
- "_model_module_version": "1.5.0",
- "_model_name": "ButtonModel",
- "_view_count": null,
- "_view_module": "@jupyter-widgets/controls",
- "_view_module_version": "1.5.0",
- "_view_name": "ButtonView",
- "button_style": "",
- "description": "Login",
- "disabled": false,
- "icon": "",
- "layout": "IPY_MODEL_ed850c64429740f5a9c624ba6a2a4060",
- "style": "IPY_MODEL_43e5ac30013846e59bd4030e794190b1",
- "tooltip": ""
- }
- },
- "a13691db6f884feeab91a60df5f0e9a7": {
- "model_module": "@jupyter-widgets/controls",
- "model_name": "HTMLModel",
- "model_module_version": "1.5.0",
- "state": {
- "_dom_classes": [],
- "_model_module": "@jupyter-widgets/controls",
- "_model_module_version": "1.5.0",
- "_model_name": "HTMLModel",
- "_view_count": null,
- "_view_module": "@jupyter-widgets/controls",
- "_view_module_version": "1.5.0",
- "_view_name": "HTMLView",
- "description": "",
- "description_tooltip": null,
- "layout": "IPY_MODEL_22524832590147ee86f0ece7e4559664",
- "placeholder": "",
- "style": "IPY_MODEL_5133470b3e744f448a94db36c881bd8b",
- "value": "\nPro Tip: If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks.