{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os; os.chdir('..')\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/ubuntu/SentenceStructureComparision/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n", "2023-10-06 11:57:13.555332: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", "2023-10-06 11:57:14.891953: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n", "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n" ] } ], "source": [ "from transformers import pipeline\n", "\n", "classifier = pipeline(\"text-classification\", model=\"models/trained_model_v11\", device=\"cuda\")" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'label': 'NEGATIVE', 'score': 0.9999995231628418}]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classifier('''Water intoxication, also known as water poisoning, hyperhydration, overhydration, or water toxemia, is a potentially fatal disturbance in brain functions that results when the normal balance of electrolytes in the body is pushed outside safe limits by excessive water intake.\n", "\n", "Under normal circumstances, accidentally consuming too much water is exceptionally rare. Nearly all deaths related to water intoxication in normal individuals have resulted either from water-drinking contests, in which individuals attempt to consume large amounts of water, or from long bouts of exercise during which excessive amounts of fluid were consumed.[1] In addition, water cure, a method of torture in which the victim is forced to consume excessive amounts of water, can cause water intoxication.[1]\n", "\n", "Water, like any other substance, can be considered a poison when over-consumed in a brief period of time. Water intoxication mostly occurs when water is being consumed in a high quantity without adequate electrolyte intake.[2]\n", "\n", "Excess of body water may also be a result of a medical condition or improper treatment; see \"hyponatremia\" for some examples. Water is considered one of the least toxic chemical compounds, with an LD50 exceeding 90 ml/kg in rats;[3] drinking six liters in three hours has caused the death of a human.[4]'''\n", "\n", ")\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'label': 'NEGATIVE', 'score': 0.9989129304885864}]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s1= '''Basketball is a team sport played by two teams of five players each. The primary objective is to score points by shooting the basketball through the opponent's hoop, which is mounted on a backboard 10 feet (3.048 meters) above the ground. The team with the most points at the end of the game wins. Basketball is played on a rectangular court, typically indoors, with a surface made of wood or synthetic materials. The rules and regulations are governed by various organizations, such as FIBA (International Basketball Federation) and the NBA (National Basketball Association). The following is a general outline of the basic rules of basketball:\n", "\n", "1. Game duration: A regulation basketball game is divided into four quarters, each lasting 12 minutes in the NBA and 10 minutes in FIBA play. College basketball in the US has two 20-minute halves. If the game is tied at the end of regulation, overtime periods are played until a winner is determined.\n", "\n", "2. Starting play: The game begins with a jump ball at the center of the court, where the referee throws the ball into the air, and one player from each team tries to gain possession by tapping it to a teammate.\n", "\n", "3. Scoring: Points are scored by shooting the ball through the hoop. A field goal made from inside the three-point arc is worth two points, while a field goal made from outside the arc is worth three points. Free throws, awarded after a foul, are worth one point each.\n", "\n", "4. Possession and dribbling: A player in possession of the ball must either pass it to a teammate or dribble (bounce) the ball while moving.'''\n", "\n", "\n", "classifier(s1)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "df= pd.read_csv(\"data/AI_checker_remade.csv\")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ParagraphAI_generated
0Bentivoglio is an Italian surname. Notable peo...0
1Stephanie McCallum (born Sydney, Australia, 3...0
2Abdāl lit: substitutes, but which can also be...1
3The Gadget is a young adult historical novel w...1
4The A-1 lifeboat was a powered lifeboat that w...0
\n", "
" ], "text/plain": [ " Paragraph AI_generated\n", "0 Bentivoglio is an Italian surname. Notable peo... 0\n", "1 Stephanie McCallum (born Sydney, Australia, 3... 0\n", "2 Abdāl lit: substitutes, but which can also be... 1\n", "3 The Gadget is a young adult historical novel w... 1\n", "4 The A-1 lifeboat was a powered lifeboat that w... 0" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Bentivoglio is an Italian surname. Notable people with the surname include: Annibale Bentivoglio (disambiguation), multiple people, including:\\nAnnibale I Bentivoglio (1415–1445), absolute ruler of the Italian city of Bologna\\nAnnibale II Bentivoglio (1467–1540), Italian condottiero who was shortly lord of Bologna\\nAnnibale Bentivoglio (archbishop) (died 1663), Italian Roman Catholic archbishop\\n Antongaleazzo Bentivoglio (c. 1385–1435), Italian condottiero who was executed by the papacy for treason\\n Cornelio Bentivoglio (1668–1732), Italian nobleman and cardinal\\n Ermes Bentivoglio (1475–1513), Italian condottiero\\n Fabrizio Bentivoglio (born 1957), Italian cinema and theatre actor and screenwriter\\n Giovanni Bentivoglio (disambiguation), multiple people, including:\\nGiovanni I Bentivoglio (died 1402), first ruler of Bologna from the Bentivoglio family\\nGiovanni II Bentivoglio (1443–1508), Italian nobleman\\n Girolamo Bentivoglio (died 1601), Roman Catholic prelate\\n Guido Bentivoglio (1579–1644), Italian cardinal, statesman and historian\\n Ippolito II Bentivoglio (1611–1685), Italian nobleman\\n Maria Francesca Bentivoglio (born 1977), Italian tennis player\\n Mary Magdalen Bentivoglio (1834–1905), Italian nun of the Order of St. Clare\\n Mirella Bentivoglio (1922–2017), Italian sculptor, poet, performance artist and curator\\n Sante Bentivoglio (1426–1462), Italian nobleman who ruled as tyrant of Bologna\\n Sean Bentivoglio (born 1985), Canadian ice hockey forward\\n Simone Bentivoglio (born 1985), Italian football midfielder\\n Vittoria Bentivoglio (16th century), Italian singer'" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s= df.iloc[[4]].Paragraph.values.tolist()[0]\n", "s= df.iloc[[0]].Paragraph.values.tolist()[0]\n", "\n", "s" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}]" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classifier(s)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ParagraphAI_generated
0Bentivoglio is an Italian surname. Notable peo...0
1Stephanie McCallum (born Sydney, Australia, 3...0
2Abdāl lit: substitutes, but which can also be...1
3The Gadget is a young adult historical novel w...1
4The A-1 lifeboat was a powered lifeboat that w...0
5D. António Tomás da Guarda Cabreira de Menezes...1
6Coirpre mac Fogartaig (died 771) was a king of...1
7Papilionanthe Miss Joaquim, also known as the ...1
8Johannes Junius (1573 – 6 August 1628) was a G...1
9KLQL (101.1 FM, \"K101\") is a radio station bro...0
\n", "
" ], "text/plain": [ " Paragraph AI_generated\n", "0 Bentivoglio is an Italian surname. Notable peo... 0\n", "1 Stephanie McCallum (born Sydney, Australia, 3... 0\n", "2 Abdāl lit: substitutes, but which can also be... 1\n", "3 The Gadget is a young adult historical novel w... 1\n", "4 The A-1 lifeboat was a powered lifeboat that w... 0\n", "5 D. António Tomás da Guarda Cabreira de Menezes... 1\n", "6 Coirpre mac Fogartaig (died 771) was a king of... 1\n", "7 Papilionanthe Miss Joaquim, also known as the ... 1\n", "8 Johannes Junius (1573 – 6 August 1628) was a G... 1\n", "9 KLQL (101.1 FM, \"K101\") is a radio station bro... 0" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head(10)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/ubuntu/SentenceStructureComparision/venv/lib/python3.10/site-packages/transformers/pipelines/base.py:1101: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999866485595703}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999996423721313}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999986886978149}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999996423721313}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999996423721313}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999970197677612}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999996423721313}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9038264155387878}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999996423721313}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999996423721313}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999759197235107}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999994039535522}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999996423721313}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999608993530273}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9998728036880493}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999996423721313}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999945163726807}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.999942421913147}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999997615814209}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n", "[{'label': 'NEGATIVE', 'score': 0.9999984502792358}] actual_value= 0\n", "\n", "[{'label': 'POSITIVE', 'score': 0.9999997615814209}] actual_value= 1\n", "\n" ] } ], "source": [ "for i in range(100):\n", " s= df.iloc[[i]].Paragraph.values.tolist()[0]\n", " actual_value= df.iloc[[i]].AI_generated.values.tolist()[0]\n", " \n", " print(classifier(s), \" \", \"actual_value= \", actual_value)\n", " # print(f\"actual_value= {actual_value}\")\n", " print()" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'label': 'POSITIVE', 'score': 0.9999997615814209}]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classifier('''Basketball is a team sport. The main agenda of the game is to score points by shooting the basketball through the opponent's hoop. The team with the most points at the end of the game wins. Basketball is played on a rectangular court surface made of wood or synthetic materials. The rules and regulations are governed by various organizations, such as FIBA (International Basketball Federation) and the NBA (National Basketball Association).''')" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/ubuntu/SentenceStructureComparision/venv/lib/python3.10/site-packages/transformers/pipelines/base.py:1101: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset\n", " warnings.warn(\n" ] }, { "data": { "text/plain": [ "[{'label': 'NEGATIVE', 'score': 0.9999992847442627}]" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classifier( '''In a quiet, picturesque village nestled deep within the lush, rolling hills of the countryside, there stood a charming, centuries-old cottage, its timeworn facade adorned with colorful flowers that cascaded down from window boxes. The cottage, with its rustic charm, had seen generations come and go, witnessed countless stories unfold within its sturdy walls. Each morning, as the sun cast its golden rays upon the sleepy hamlet, the villagers would wake to the melodious chirping of birds, their cheerful songs serving as a gentle alarm clock. Life in the village was slow-paced, a stark contrast to the bustling cities with their constant noise and ceaseless activity. Time seemed to move differently here, as if the world beyond the village's borders existed in a parallel universe, always in a hurry, while the village embraced a rhythm that ebbed and flowed with the changing seasons. The villagers, bound by a strong sense of community, gathered for festivals, sharing laughter and stories around bonfires that crackled in the cool night air. Generations of families had lived in this idyllic haven, passing down stories, traditions, and the enduring spirit of the village from one age to the next, ensuring that the passage of time only deepened their connection to this place they called home.''')" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'label': 'POSITIVE', 'score': 0.9999997615814209}]" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classifier('''Political stability is the ability of a government to maintain order and authority within its borders. It is essential for economic growth, as it provides a foundation for investment and trade.\n", "There are many factors that contribute to political stability, including:\n", "A strong rule of law: The rule of law is the principle that everyone is subject to the same laws, regardless of their social status or political affiliation. A strong rule of law helps to prevent corruption and ensures that everyone has equal opportunity to succeed.\n", "A well-functioning government: A well-functioning government is one that is able to provide essential services, such as security, education, and healthcare. It is also able to manage the economy effectively and to respond to crises.\n", "A vibrant civil society: A vibrant civil society is one that is made up of active and engaged citizens. Civil society organizations can help to hold the government accountable and to promote democracy and good governance.\n", "Political stability is not always easy to achieve, but it is essential for economic growth. By investing in political stability, we can create a foundation for long-term prosperity.\n", "Here are some of the benefits of political stability:''')" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/ubuntu/SentenceStructureComparision/venv/lib/python3.10/site-packages/transformers/pipelines/base.py:1101: UserWarning: You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset\n", " warnings.warn(\n" ] }, { "data": { "text/plain": [ "[{'label': 'NEGATIVE', 'score': 0.9713687300682068}]" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "classifier('''12\n", "\n", "Yes, the perplexity is always equal to two to the power of the entropy. It doesn't matter what type of model you have, n-gram, unigram, or neural network.\n", "\n", "There are a few reasons why language modeling people like perplexity instead of just using entropy. One is that, because of the exponent, improvements in perplexity \"feel\" like they are more substantial than the equivalent improvement in entropy. Another is that before they started using perplexity, the complexity of a language model was reported using a simplistic branching factor measurement that is more similar to perplexity than it is to entropy.''')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 2 }