notebooks_explicativos/Estatistico.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lawNHLqffR_m"
      },
      "source": [
        "# SCC0633/SCC5908 - Processamento de Linguagem Natural\n",
        "> **Docente:** Thiago Alexandre Salgueiro Pardo \\\n",
        "> **Estagiário PAE:** Germano Antonio Zani Jorge\n",
        "\n",
        "\n",
        "# Integrantes do Grupo: GPTrouxas\n",
        "> André Guarnier De Mitri - 11395579 \\\n",
        "> Daniel Carvalho - 10685702 \\\n",
        "> Fernando - 11795342 \\\n",
        "> Lucas Henrique Sant'Anna - 10748521 \\\n",
        "> Magaly L Fujimoto - 4890582"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pV6WGoBln8id"
      },
      "source": [
        "# New Section"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Abordagem Estatístico\n",
        "A arquitetura da solução estatística/neural envolve duas abordagens que\n",
        "serão descritas neste documento. A primeira abordagem envolve utilizar\n",
        "TF-IDF e Naive Bayes. E a segunda abordagem irá utilizar Word2Vec e um\n",
        "modelo transformers pré-treinado da família BERT, realizando finetuning do\n",
        "modelo.\n",
        "\n",
        "Na primeira abordagem, utilizaremos o TF-IDF, que leva em consideração a\n",
        "frequência de ocorrência dos termos em um corpus e gera uma sequência de\n",
        "vetores que serão fornecidos ao Naive Bayes para classificação da review como\n",
        "positiva ou negativa.\n",
        "\n",
        "\n",
        "Na segunda abordagem, utilizaremos o Word2Vec para vetorizar as reviews.\n",
        "Após dividir em treino e teste, faremos o fine tuning de um modelo do tipo BERT\n",
        "para o nosso problema e dataset específico. Com o BERT adaptado, faremos a\n",
        "classificação de nossos textos, medindo o seu desempenho com F1 score e\n",
        "acurácia.\n",
        "\n",
        "![alt text](../imagens/BERT_TDIDF.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vfP54aryxZBg"
      },
      "source": [
        "\n",
        "## # Etapas da Abordagem Estatística\n",
        "\n",
        "1. **Bibliotecas**: Importamos as bibliotecas necessárias, considerando pandas para manipulação de dados, train_test_split para dividir o conjunto de dados em conjuntos de treinamento e teste, TfidfVectorizer para vetorização de texto usando TF-IDF, MultinomialNB para implementar o classificador Naive Bayes Multinomial e algumas métricas de avaliação.\n",
        "\n",
        "2. **Conjunto de dados**: Carregar o conjunto de dados e armazená-lo em um dataframe usando pandas.\n",
        "\n",
        "3. **Dividir o conjunto de dados**: Usamos `train_test_split` para dividir o DataFrame em conjuntos de treinamento e teste.\n",
        "\n",
        "4. **TF-IDF**: Usamos `TfidfVectorizer` para converter as revisões de texto em vetores numéricos usando a técnica TF-IDF. Em seguida, ajustamos e transformamos tanto o conjunto de treinamento quanto o conjunto de teste.\n",
        "\n",
        "5. **Naive Bayes**: Treinamos um classificador Naive Bayes Multinomial e usamos o modelo treinado para prever os sentimentos no conjunto de teste usando `predict`.\n",
        "\n",
        "6. **Avaliação e Resultados**: Salvamos os resultados em um novo dataframe `results_df` contendo as revisões do conjunto de teste, os sentimentos originais e os sentimentos previstos pelo modelo. Além disso, avaliamos o modelo verificando algumas métricas e a matriz de confusão.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TbLraa4UhWDJ"
      },
      "source": [
        "\n",
        "## # Baixando, Carregando os dados e Pré Processamento\n",
        "\n",
        "1. Transformar todos os textos em lowercase \\\\\n",
        "2. Remoção de caracteres especiais \\\\\n",
        "3. Remoção de stop words \\\\\n",
        "4. Lematização (Lemmatization) \\\\\n",
        "5. Tokenização \\\\"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "id": "bIWmIe0qfTbE"
      },
      "outputs": [],
      "source": [
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "Wf0n2yPdAn4C",
        "outputId": "37eb3c4d-40c1-41a0-9b1a-d93ed6e272f3"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"db\",\n  \"rows\": 50000,\n  \"fields\": [\n    {\n      \"column\": \"review\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 49582,\n        \"samples\": [\n          \"\\\"Soul Plane\\\" is a horrible attempt at comedy that only should appeal people with thick skulls, bloodshot eyes and furry pawns. <br /><br />The plot is not only incoherent but also non-existent, acting is mostly sub sub-par with a gang of highly moronic and dreadful characters thrown in for bad measure, jokes are often spotted miles ahead and almost never even a bit amusing. This movie lacks any structure and is full of racial stereotypes that must have seemed old even in the fifties, the only thing it really has going for it is some pretty ladies, but really, if you want that you can rent something from the \\\"Adult\\\" section. OK?<br /><br />I can hardly see anything here to recommend since you'll probably have a lot a better and productive time chasing rats with a sledgehammer or inventing waterproof teabags or whatever.<br /><br />2/10\",\n          \"Guest from the Future tells a fascinating story of time travel, friendship, battle of good and evil -- all with a small budget, child actors, and few special effects. Something for Spielberg and Lucas to learn from. ;) A sixth-grader Kolya \\\"Nick\\\" Gerasimov finds a time machine in the basement of a decrepit building and travels 100 years into the future. He discovers a near-perfect, utopian society where robots play guitars and write poetry, everyone is kind to each other and people enjoy everything technology has to offer. Alice is the daughter of a prominent scientist who invented a device called Mielophone that allows to read minds of humans and animals. The device can be put to both good and bad use, depending on whose hands it falls into. When two evil space pirates from Saturn who want to rule the universe attempt to steal Mielophone, it falls into the hands of 20th century school boy Nick. With the pirates hot on his tracks, he travels back to his time, followed by the pirates, and Alice. Chaos, confusion and funny situations follow as the luckless pirates try to blend in with the earthlings. Alice enrolls in the same school Nick goes to and demonstrates superhuman abilities in PE class. The catch is, Alice doesn't know what Nick looks like, while the pirates do. Also, the pirates are able to change their appearance and turn literally into anyone. (Hmm, I wonder if this is where James Cameron got the idea for Terminator...) Who gets to Nick -- and Mielophone -- first? Excellent plot, non-stop adventures, and great soundtrack. I wish Hollywood made kid movies like this one...\",\n          \"\\\"National Treasure\\\" (2004) is a thoroughly misguided hodge-podge of plot entanglements that borrow from nearly every cloak and dagger government conspiracy clich\\u00e9 that has ever been written. The film stars Nicholas Cage as Benjamin Franklin Gates (how precious is that, I ask you?); a seemingly normal fellow who, for no other reason than being of a lineage of like-minded misguided fortune hunters, decides to steal a 'national treasure' that has been hidden by the United States founding fathers. After a bit of subtext and background that plays laughably (unintentionally) like Indiana Jones meets The Patriot, the film degenerates into one misguided whimsy after another \\u0096 attempting to create a 'Stanley Goodspeed' regurgitation of Nicholas Cage and launch the whole convoluted mess forward with a series of high octane, but disconnected misadventures.<br /><br />The relevancy and logic to having George Washington and his motley crew of patriots burying a king's ransom someplace on native soil, and then, going through the meticulous plan of leaving clues scattered throughout U.S. currency art work, is something that director Jon Turteltaub never quite gets around to explaining. Couldn't Washington found better usage for such wealth during the start up of the country? Hence, we are left with a mystery built on top of an enigma that is already on shaky ground by the time Ben appoints himself the new custodian of this untold wealth. Ben's intentions are noble \\u0096 if confusing. He's set on protecting the treasure. For who and when?\\u0085your guess is as good as mine.<br /><br />But there are a few problems with Ben's crusade. First up, his friend, Ian Holmes (Sean Bean) decides that he can't wait for Ben to make up his mind about stealing the Declaration of Independence from the National Archives (oh, yeah \\u0096 brilliant idea!). Presumably, the back of that famous document holds the secret answer to the ultimate fortune. So Ian tries to kill Ben. The assassination attempt is, of course, unsuccessful, if overly melodramatic. It also affords Ben the opportunity to pick up, and pick on, the very sultry curator of the archives, Abigail Chase (Diane Kruger). She thinks Ben is clearly a nut \\u0096 at least at the beginning. But true to action/romance form, Abby's resolve melts quicker than you can say, \\\"is that the Hope Diamond?\\\" The film moves into full X-File-ish mode, as the FBI, mistakenly believing that Ben is behind the theft, retaliate in various benign ways that lead to a multi-layering of action sequences reminiscent of Mission Impossible meets The Fugitive. Honestly, don't those guys ever get 'intelligence' information that is correct? In the final analysis, \\\"National Treasure\\\" isn't great film making, so much as it's a patchwork rehash of tired old bits from other movies, woven together from scraps, the likes of which would make IL' Betsy Ross blush.<br /><br />The Buena Vista DVD delivers a far more generous treatment than this film is deserving of. The anamorphic widescreen picture exhibits a very smooth and finely detailed image with very rich colors, natural flesh tones, solid blacks and clean whites. The stylized image is also free of blemishes and digital enhancements. The audio is 5.1 and delivers a nice sonic boom to your side and rear speakers with intensity and realism. Extras include a host of promotional junket material that is rather deep and over the top in its explanation of how and why this film was made. If only, as an audience, we had had more clarification as to why Ben and co. were chasing after an illusive treasure, this might have been one good flick. Extras conclude with the theatrical trailer, audio commentary and deleted scenes. Not for the faint-hearted \\u0096 just the thick-headed.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentiment\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"negative\",\n          \"positive\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "db"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-2cb947b9-c799-42cf-8e37-5ee6fe2a7cec\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>One of the other reviewers has mentioned that ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>A wonderful little production. &lt;br /&gt;&lt;br /&gt;The...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>I thought this was a wonderful way to spend ti...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Basically there's a family where a little boy ...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Petter Mattei's \"Love in the Time of Money\" is...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2cb947b9-c799-42cf-8e37-5ee6fe2a7cec')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-2cb947b9-c799-42cf-8e37-5ee6fe2a7cec button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-2cb947b9-c799-42cf-8e37-5ee6fe2a7cec');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-711a5547-60a9-42cc-80f0-6083c4c007ee\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-711a5547-60a9-42cc-80f0-6083c4c007ee')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-711a5547-60a9-42cc-80f0-6083c4c007ee button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                              review sentiment\n",
              "0  One of the other reviewers has mentioned that ...  positive\n",
              "1  A wonderful little production. <br /><br />The...  positive\n",
              "2  I thought this was a wonderful way to spend ti...  positive\n",
              "3  Basically there's a family where a little boy ...  negative\n",
              "4  Petter Mattei's \"Love in the Time of Money\" is...  positive"
            ]
          },
          "execution_count": 52,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "db = pd.read_csv('imdb_reviews.csv')\n",
        "db.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6PlfPScGMF1_",
        "outputId": "2a0bd4a1-e22a-429d-82a4-5984eeab7b9d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "sentiment\n",
              "positive    25000\n",
              "negative    25000\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 53,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "db['sentiment'].value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kev0EaSmMa4N",
        "outputId": "eab73a61-ba36-4d72-e4f2-82236f9f2880"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quantidade de valores faltantes para cada variável do dataset:\n",
            "review       0\n",
            "sentiment    0\n",
            "dtype: int64\n"
          ]
        }
      ],
      "source": [
        "valores_ausentes = db.isnull().sum(axis=0)\n",
        "print('Quantidade de valores faltantes para cada variável do dataset:')\n",
        "print(valores_ausentes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 276
        },
        "id": "1AI3rN0KMuUq",
        "outputId": "7ea5c91b-362e-49eb-82a7-6e8535f0e591"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"db\",\n  \"rows\": 50000,\n  \"fields\": [\n    {\n      \"column\": \"review\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 49574,\n        \"samples\": [\n          \"moving intriguing absorbing however story little choppy hard follow time although two principal actor great job seeing senn penn acting every fiber stealing every frame made memorable movie later movie revealed one role actor also showed comedic flair sweet lowdown surprisingly talented light weight used think \",\n          \"gem go direct video fabulous art direction mood never miss beat truman show meet metropolis excellent cast never seen laura dern better bill macy always fabulous said david paymer meat loaf incredible film \",\n          \"watched movie dismayed say least movie failed communicate audience language would put shame street loafer plot father forcing none son marry seems far fetched idea grandmother asking grand kid mess enemy would draw feeble minded attention waiting whole movie laugh laugh stupidity waste 3 hour convince movie even worth first look hope save time \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentiment\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"negative\",\n          \"positive\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "db"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-5fddbeb1-c284-468b-b861-38e8e649d721\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>one reviewer mentioned watching 1 oz episode h...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>wonderful little production filming technique ...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>thought wonderful way spend time hot summer we...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>basically family little boy jake think zombie ...</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>petter mattei love time money visually stunnin...</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5fddbeb1-c284-468b-b861-38e8e649d721')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5fddbeb1-c284-468b-b861-38e8e649d721 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5fddbeb1-c284-468b-b861-38e8e649d721');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-430854d7-7cfd-4ac6-9592-bec216bf2654\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-430854d7-7cfd-4ac6-9592-bec216bf2654')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-430854d7-7cfd-4ac6-9592-bec216bf2654 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                              review sentiment\n",
              "0  one reviewer mentioned watching 1 oz episode h...  positive\n",
              "1  wonderful little production filming technique ...  positive\n",
              "2  thought wonderful way spend time hot summer we...  positive\n",
              "3  basically family little boy jake think zombie ...  negative\n",
              "4  petter mattei love time money visually stunnin...  positive"
            ]
          },
          "execution_count": 55,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import re\n",
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import PorterStemmer\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "\n",
        "def lowercase_text(text):\n",
        "    return text.lower()\n",
        "\n",
        "def remove_html(text):\n",
        "    return re.sub(r'<[^<]+?>', '', text)\n",
        "\n",
        "def remove_url(text):\n",
        "    return re.sub(r'http[s]?://\\S+|www\\.\\S+', '', text)\n",
        "\n",
        "def remove_punctuations(text):\n",
        "    tokens_list = '!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'\n",
        "    for char in text:\n",
        "        if char in tokens_list:\n",
        "            text = text.replace(char, ' ')\n",
        "\n",
        "    return text\n",
        "\n",
        "def remove_emojis(text):\n",
        "    emojis = re.compile(\"[\"\n",
        "                        u\"\\U0001F600-\\U0001F64F\"\n",
        "                        u\"\\U0001F300-\\U0001F5FF\"\n",
        "                        u\"\\U0001F680-\\U0001F6FF\"\n",
        "                        u\"\\U0001F1E0-\\U0001F1FF\"\n",
        "                        u\"\\U00002500-\\U00002BEF\"\n",
        "                        u\"\\U00002702-\\U000027B0\"\n",
        "                        u\"\\U00002702-\\U000027B0\"\n",
        "                        u\"\\U000024C2-\\U0001F251\"\n",
        "                        u\"\\U0001f926-\\U0001f937\"\n",
        "                        u\"\\U00010000-\\U0010ffff\"\n",
        "                        u\"\\u2640-\\u2642\"\n",
        "                        u\"\\u2600-\\u2B55\"\n",
        "                        u\"\\u200d\"\n",
        "                        u\"\\u23cf\"\n",
        "                        u\"\\u23e9\"\n",
        "                        u\"\\u231a\"\n",
        "                        u\"\\ufe0f\"\n",
        "                        u\"\\u3030\"\n",
        "                        \"]+\", re.UNICODE)\n",
        "\n",
        "    text = re.sub(emojis, '', text)\n",
        "    return text\n",
        "\n",
        "def remove_stop_words(text):\n",
        "    stop_words = stopwords.words('english')\n",
        "    new_text = ''\n",
        "    for word in text.split():\n",
        "        if word not in stop_words:\n",
        "            new_text += ''.join(f'{word} ')\n",
        "\n",
        "    return new_text.strip()\n",
        "\n",
        "def lem_words(text):\n",
        "    lemma = WordNetLemmatizer()\n",
        "    new_text = ''\n",
        "    for word in text.split():\n",
        "        new_text += ''.join(f'{lemma.lemmatize(word)} ')\n",
        "\n",
        "    return new_text\n",
        "\n",
        "def preprocess_text(text):\n",
        "    text = lowercase_text(text)\n",
        "    text = remove_html(text)\n",
        "    text = remove_url(text)\n",
        "    text = remove_punctuations(text)\n",
        "    text = remove_emojis(text)\n",
        "    text = remove_stop_words(text)\n",
        "    text = lem_words(text)\n",
        "\n",
        "    return text\n",
        "\n",
        "nltk.download('stopwords')\n",
        "nltk.download('wordnet')\n",
        "db['review'] = db['review'].apply(preprocess_text)\n",
        "db.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QgufZpgHnPa4"
      },
      "source": [
        "# **Conjunto de Treino e teste**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {
        "id": "s0lJ6Q0tnPka"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X= db['review']\n",
        "y= db['sentiment']\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state= 12)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nz4erCEJuD4-",
        "outputId": "88d57536-66e7-4d9b-e016-bf40183d4c45"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "35235    disagree people saying lousy horror film good ...\n",
              "36936    husband wife doctor team carole nile nelson mo...\n",
              "46486    like cast pretty much however story sort unfol...\n",
              "27160    movie awful bad bear expend anything word avoi...\n",
              "19490    purchased blood castle dvd ebay buck knowing s...\n",
              "                               ...                        \n",
              "36482    strange thing see film scene work rather weakl...\n",
              "40177    saw cheap dvd release title entity force since...\n",
              "19709    one peculiar oft used romance movie plot one s...\n",
              "38555    nothing positive say meandering nonsense huffi...\n",
              "14155    low moment life bewildered depressed sitting r...\n",
              "Name: review, Length: 40000, dtype: object"
            ]
          },
          "execution_count": 57,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_train"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6LX-6e-QlioJ"
      },
      "source": [
        "# **TD-IDF e Naive Bayes**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "metadata": {
        "id": "gscB9-obNusA"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import confusion_matrix,classification_report\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.preprocessing import StandardScaler as encoder\n",
        "from sklearn.metrics import (\n",
        "    accuracy_score,\n",
        "    confusion_matrix,\n",
        "    ConfusionMatrixDisplay,\n",
        "    f1_score,\n",
        ")\n",
        "\n",
        "\n",
        "tfidf = TfidfVectorizer()\n",
        "tfidf_train = tfidf.fit_transform(X_train)\n",
        "tfidf_test = tfidf.transform(X_test)\n",
        "\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "\n",
        "naive_bayes = MultinomialNB()\n",
        "\n",
        "naive_bayes.fit(tfidf_train, y_train)\n",
        "y_pred = naive_bayes.predict(tfidf_test)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "RfJ7AHMZvAb8",
        "outputId": "685701e1-b1e8-47fb-9dc5-1bc04dd3894b"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"results_df\",\n  \"rows\": 10000,\n  \"fields\": [\n    {\n      \"column\": \"review\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 9990,\n        \"samples\": [\n          \"saw lot film charles dickens christmas carol one best atmosphere exactly book actor george c scott others great unfortunately often watch film germany switzerland \",\n          \"loved first season quality went little bit second season however great middle pegasus third season fairly novel original ok fourth season started going downhill fast never even began giving u explanation really starting need hell cylon plan two cylon faction point angel kara leading fleet devastated earth 1 kind past last five cylons survive reincarnation question everywhere answer nowhere come end earth 2 earth past well okay destroying fleet giving technology giving kind urban life spreading thousand people paper thinly across planet anti science anti reason anti life philosophy show seems humanity forever trapped cycle going nature romanticism decadent capitalist society inventing destructive ruin everything without vision without hope grander future humanity antithetical proper science fiction even get started angel religious claptrap worst kind ultimate disappointment whole happened happen thing related previous incarnation series earth know making new show somehow consistent old would definitive stroke genius frakkin shame 1 10 \",\n          \"guest future tell fascinating story time travel friendship battle good evil small budget child actor special effect something spielberg lucas learn sixth grader kolya nick gerasimov find time machine basement decrepit building travel 100 year future discovers near perfect utopian society robot play guitar write poetry everyone kind people enjoy everything technology offer alice daughter prominent scientist invented device called mielophone allows read mind human animal device put good bad use depending whose hand fall two evil space pirate saturn want rule universe attempt steal mielophone fall hand 20th century school boy nick pirate hot track travel back time followed pirate alice chaos confusion funny situation follow luckless pirate try blend earthling alice enrolls school nick go demonstrates superhuman ability pe class catch alice know nick look like pirate also pirate able change appearance turn literally anyone hmm wonder james cameron got idea terminator get nick mielophone first excellent plot non stop adventure great soundtrack wish hollywood made kid movie like one \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"original sentiment\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"positive\",\n          \"negative\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"predicted sentiment\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"positive\",\n          \"negative\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "results_df"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-dc3f6f1f-8cfd-42e3-89f7-04a33171d9a5\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>original sentiment</th>\n",
              "      <th>predicted sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>34622</th>\n",
              "      <td>hard tell noonan marshall trying ape abbott co...</td>\n",
              "      <td>negative</td>\n",
              "      <td>negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1163</th>\n",
              "      <td>well start one reviewer said know real treat s...</td>\n",
              "      <td>positive</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7637</th>\n",
              "      <td>wife kid opinion absolute abc classic seen eve...</td>\n",
              "      <td>positive</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7045</th>\n",
              "      <td>surprise basic copycat comedy classic nutty pr...</td>\n",
              "      <td>positive</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43847</th>\n",
              "      <td>josef von sternberg directs magnificent silent...</td>\n",
              "      <td>positive</td>\n",
              "      <td>positive</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dc3f6f1f-8cfd-42e3-89f7-04a33171d9a5')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-dc3f6f1f-8cfd-42e3-89f7-04a33171d9a5 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-dc3f6f1f-8cfd-42e3-89f7-04a33171d9a5');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-399174c9-298e-4ffd-acce-3dabb38c956b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-399174c9-298e-4ffd-acce-3dabb38c956b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-399174c9-298e-4ffd-acce-3dabb38c956b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                  review original sentiment  \\\n",
              "34622  hard tell noonan marshall trying ape abbott co...           negative   \n",
              "1163   well start one reviewer said know real treat s...           positive   \n",
              "7637   wife kid opinion absolute abc classic seen eve...           positive   \n",
              "7045   surprise basic copycat comedy classic nutty pr...           positive   \n",
              "43847  josef von sternberg directs magnificent silent...           positive   \n",
              "\n",
              "      predicted sentiment  \n",
              "34622            negative  \n",
              "1163             positive  \n",
              "7637             positive  \n",
              "7045             positive  \n",
              "43847            positive  "
            ]
          },
          "execution_count": 59,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Criando DataFrame com resultados\n",
        "results_df = pd.DataFrame({'review': X_test, 'original sentiment': y_test, 'predicted sentiment': y_pred})\n",
        "results_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Xq2ABXYtsjk"
      },
      "source": [
        "## Avaliação"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "id": "3lXqDNhSrhsZ"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import confusion_matrix, classification_report\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_confusion_matrix(y_true, y_pred, labels, model_name):\n",
        "    cm = confusion_matrix(y_true, y_pred, labels=labels)\n",
        "    plt.figure(figsize=(8, 6))\n",
        "    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=labels, yticklabels=labels)\n",
        "    plt.xlabel('Predicted Labels')\n",
        "    plt.ylabel('True Labels')\n",
        "    plt.title(f'Confusion Matrix {model_name}')\n",
        "    plt.show()\n",
        "\n",
        "# Função para calcular e imprimir as métricas de avaliação\n",
        "def print_evaluation_metrics(y_true, y_pred, model_name):\n",
        "    print(f\"Classification Report {model_name}:\")\n",
        "    print(classification_report(y_true, y_pred))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "id": "ybfb_GKDuqmb",
        "outputId": "3e4c3a98-8962-4ce8-9856-2252f769a1b8"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAApIAAAIjCAYAAACwHvu2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABj0UlEQVR4nO3deVxV1f7/8fdB5YAgkwpozkMmqTmVoeWQA06lZbecEuerF0dyyNIcyujadSpLy0rNtDJLK62UVDQTh1Qci9RQLAWcEFFEhf37w5/n2xFN2Hk8R3g9fezH5ay99t6ffR4X/fRZa69tMQzDEAAAAJBHbs4OAAAAAHcnEkkAAACYQiIJAAAAU0gkAQAAYAqJJAAAAEwhkQQAAIApJJIAAAAwhUQSAAAAppBIAgAAwBQSSaCAOHDggFq1aiVfX19ZLBYtX778tp7/8OHDslgsmj9//m09792sadOmatq0qbPDAACHIZEE7qBDhw7p3//+typVqiQPDw/5+PioUaNGmjlzpjIyMhx67fDwcO3Zs0eTJ0/WwoULVb9+fYde707q2bOnLBaLfHx8bvg9HjhwQBaLRRaLRf/73//yfP5jx45pwoQJiouLuw3RmnftHqZOnZpj3/z582WxWPTzzz/b2iZMmGA7xmKxyM3NTaVKlVL79u21efPmOxk6gHyqsLMDAAqKlStX6l//+pesVqt69OihGjVq6NKlS9q4caNGjhypffv26b333nPItTMyMhQbG6uXXnpJgwYNcsg1ypcvr4yMDBUpUsQh57+VwoUL68KFC/rmm2/0zDPP2O1btGiRPDw8dPHiRVPnPnbsmCZOnKgKFSqodu3auT5u9erVpq53K2+88YYGDhyookWL5qr/7Nmz5e3trezsbB09elRz585V48aNtXXr1jzdDwBcj0QSuAMSEhLUuXNnlS9fXmvXrlWpUqVs+yIiInTw4EGtXLnSYdc/ceKEJMnPz89h17BYLPLw8HDY+W/FarWqUaNG+uSTT3IkkosXL1a7du30xRdf3JFYLly4oKJFi8rd3f22n7t27dqKi4vTnDlzFBkZmatjnn76aZUoUcL2uWPHjqpRo4Y+//xzEkkA/whD28AdMGXKFKWnp+uDDz6wSyKvqVKlioYOHWr7fOXKFb3yyiuqXLmyrFarKlSooBdffFGZmZl2x1WoUEHt27fXxo0b9dBDD8nDw0OVKlXSRx99ZOszYcIElS9fXpI0cuRIWSwWVahQQdLVIeFrP//VtSHRv4qOjtYjjzwiPz8/eXt7q1q1anrxxRdt+282R3Lt2rV69NFH5eXlJT8/P3Xo0EG//PLLDa938OBB9ezZU35+fvL19VWvXr104cKFm3+x1+natau+++47paam2tq2bdumAwcOqGvXrjn6nz59WiNGjFDNmjXl7e0tHx8ftWnTRrt27bL1iYmJ0YMPPihJ6tWrl22Y+Np9Nm3aVDVq1ND27dvVuHFjFS1a1Pa9XD9HMjw8XB4eHjnuPywsTP7+/jp27Ngt77FRo0Z67LHHNGXKFNPTIYKDgyVdreICwD9BIgncAd98840qVaqkhg0b5qp/37599fLLL6tu3bqaPn26mjRpoqioKHXu3DlH34MHD+rpp59Wy5YtNXXqVPn7+6tnz57at2+fJOmpp57S9OnTJUldunTRwoULNWPGjDzFv2/fPrVv316ZmZmaNGmSpk6dqieeeEI//fTT3x73ww8/KCwsTCkpKZowYYIiIyO1adMmNWrUSIcPH87R/5lnntG5c+cUFRWlZ555RvPnz9fEiRNzHedTTz0li8WiL7/80ta2ePFi3Xfffapbt26O/r///ruWL1+u9u3ba9q0aRo5cqT27NmjJk2a2JK66tWra9KkSZKk/v37a+HChVq4cKEaN25sO8+pU6fUpk0b1a5dWzNmzFCzZs1uGN/MmTNVsmRJhYeHKysrS5L07rvvavXq1XrrrbdUunTpXN3nhAkTlJycrNmzZ+eq/+nTp3Xy5EmlpKRo586d6tevnzw8PHJUbgEgzwwADnX27FlDktGhQ4dc9Y+LizMkGX379rVrHzFihCHJWLt2ra2tfPnyhiRjw4YNtraUlBTDarUazz//vK0tISHBkGS88cYbducMDw83ypcvnyOG8ePHG3/962H69OmGJOPEiRM3jfvaNebNm2drq127thEYGGicOnXK1rZr1y7Dzc3N6NGjR47r9e7d2+6cTz75pFG8ePGbXvOv9+Hl5WUYhmE8/fTTRvPmzQ3DMIysrCwjODjYmDhx4g2/g4sXLxpZWVk57sNqtRqTJk2ytW3bti3HvV3TpEkTQ5IxZ86cG+5r0qSJXduqVasMScarr75q/P7774a3t7fRsWPHW96jYRiGJCMiIsIwDMNo1qyZERwcbFy4cMEwDMOYN2+eIcnYtm2brf+17/X6zc/Pz/j+++9zdU0A+DtUJAEHS0tLkyQVK1YsV/2//fZbScox/+3555+XpBxzKUNCQvToo4/aPpcsWVLVqlXT77//bjrm612bW/nVV18pOzs7V8ccP35ccXFx6tmzpwICAmzttWrVUsuWLW33+VcDBgyw+/zoo4/q1KlTtu8wN7p27aqYmBglJSVp7dq1SkpKuuGwtnR1XqWb29W/BrOysnTq1CnbsP2OHTtyfU2r1apevXrlqm+rVq3073//W5MmTdJTTz0lDw8Pvfvuu7m+1jUTJkxQUlKS5syZc8u+X3zxhaKjo7V69WrNmzdP9957rzp16qRNmzbl+boA8FckkoCD+fj4SJLOnTuXq/5HjhyRm5ubqlSpYtceHBwsPz8/HTlyxK69XLlyOc7h7++vM2fOmIw4p2effVaNGjVS3759FRQUpM6dO2vJkiV/m1Rei7NatWo59lWvXl0nT57U+fPn7dqvvxd/f39JytO9tG3bVsWKFdNnn32mRYsW6cEHH8zxXV6TnZ2t6dOnq2rVqrJarSpRooRKliyp3bt36+zZs7m+5j333JOnB2v+97//KSAgQHFxcXrzzTcVGBiY62Ovady4sZo1a5aruZKNGzdWixYt1LJlS/Xs2VNr1qxRsWLFNHjw4DxfFwD+ikQScDAfHx+VLl1ae/fuzdNx1z/scjOFChW6YbthGKavcW3+3jWenp7asGGDfvjhBz333HPavXu3nn32WbVs2TJH33/in9zLNVarVU899ZQWLFigZcuW3bQaKUmvvfaaIiMj1bhxY3388cdatWqVoqOjdf/99+e68ipd/X7yYufOnUpJSZEk7dmzJ0/H/tX48eOVlJSU54qmt7e3GjRooB07duRI5gEgL0gkgTugffv2OnTokGJjY2/Zt3z58srOztaBAwfs2pOTk5Wammp7Avt28Pf3t3vC+Zrrq56S5ObmpubNm2vatGnav3+/Jk+erLVr12rdunU3PPe1OOPj43Ps+/XXX1WiRAl5eXn9sxu4ia5du2rnzp06d+7cDR9Qumbp0qVq1qyZPvjgA3Xu3FmtWrVSixYtcnwnuU3qc+P8+fPq1auXQkJC1L9/f02ZMkXbtm0zda4mTZqoadOm+u9//5vnJ7ivXLkiSUpPTzd1bQCQSCSBO2LUqFHy8vJS3759lZycnGP/oUOHNHPmTElXh2Yl5Xiyetq0aZKkdu3a3ba4KleurLNnz2r37t22tuPHj2vZsmV2/U6fPp3j2GvrD16/JNE1pUqVUu3atbVgwQK7xGzv3r1avXq17T4doVmzZnrllVc0a9Ys21I3N1KoUKEc1c7PP/9cf/75p13btYT3Rkl3Xo0ePVqJiYlasGCBpk2bpgoVKig8PPym3+OtXJsrmZfF7E+fPq1NmzYpODjY1LA6AFzDImLAHVC5cmUtXrxYzz77rKpXr273ZptNmzbp888/V8+ePSVJDzzwgMLDw/Xee+8pNTVVTZo00datW7VgwQJ17NjxpkvLmNG5c2eNHj1aTz75pIYMGaILFy5o9uzZuvfee+0eNpk0aZI2bNigdu3aqXz58kpJSdE777yjMmXK6JFHHrnp+d944w21adNGoaGh6tOnjzIyMvTWW2/J19dXEyZMuG33cT03NzeNHTv2lv3at2+vSZMmqVevXmrYsKH27NmjRYsWqVKlSnb9KleuLD8/P82ZM0fFihWTl5eXGjRooIoVK+YprrVr1+qdd97R+PHjbcsRzZs3T02bNtW4ceM0ZcqUPJ1PulqVbNKkidavX3/TPkuXLpW3t7cMw9CxY8f0wQcf6MyZM5ozZ85trbYCKICc+9A4ULD89ttvRr9+/YwKFSoY7u7uRrFixYxGjRoZb731lnHx4kVbv8uXLxsTJ040KlasaBQpUsQoW7asMWbMGLs+hnF1+Z927drluM71y87cbPkfwzCM1atXGzVq1DDc3d2NatWqGR9//HGO5X/WrFljdOjQwShdurTh7u5ulC5d2ujSpYvx22+/5bjG9Uvk/PDDD0ajRo0MT09Pw8fHx3j88ceN/fv32/W5dr3rlxe6tqRNQkLCTb9Tw7Bf/udmbrb8z/PPP2+UKlXK8PT0NBo1amTExsbecNmer776yggJCTEKFy5sd59NmjQx7r///hte86/nSUtLM8qXL2/UrVvXuHz5sl2/4cOHG25ubkZsbOzf3oP+svzPX61bt862tM+tlv/x8vIyQkNDjSVLlvzttQAgNyyGkYdZ7AAAAMD/xxxJAAAAmEIiCQAAAFNIJAEAAGAKiSQAAABMIZEEAACAKSSSAAAAMIVEEgAAAKbkyzfbeDYY6ewQADjIyR/z/vYXAHcHL3fnvWnJs84gh507Y+csh53b2ahIAgAAwJR8WZEEAADIEwu1NTNIJAEAACzOG1a/m5F+AwAAwBQqkgAAAAxtm8K3BgAAAFOoSAIAADBH0hQqkgAAADCFiiQAAABzJE3hWwMAAIApVCQBAACYI2kKiSQAAABD26bwrQEAAMAUKpIAAAAMbZtCRRIAAACmUJEEAABgjqQpfGsAAAAwhYokAAAAcyRNoSIJAAAAU6hIAgAAMEfSFBJJAAAAhrZNIf0GAACAKVQkAQAAGNo2hW8NAAAAplCRBAAAoCJpCt8aAAAATKEiCQAA4MZT22ZQkQQAAIApVCQBAACYI2kKiSQAAAALkptC+g0AAABTqEgCAAAwtG0K3xoAAABMoSIJAADAHElTqEgCAADAFCqSAAAAzJE0hW8NAAAAplCRBAAAYI6kKSSSAAAADG2bwrcGAAAAU0gkAQAALBbHbf/A66+/LovFomHDhtnaLl68qIiICBUvXlze3t7q1KmTkpOT7Y5LTExUu3btVLRoUQUGBmrkyJG6cuWKXZ+YmBjVrVtXVqtVVapU0fz58/McH4kkAACAC9q2bZveffdd1apVy659+PDh+uabb/T5559r/fr1OnbsmJ566inb/qysLLVr106XLl3Spk2btGDBAs2fP18vv/yyrU9CQoLatWunZs2aKS4uTsOGDVPfvn21atWqPMVIIgkAAGBxc9xmQnp6urp166a5c+fK39/f1n727Fl98MEHmjZtmh577DHVq1dP8+bN06ZNm7R582ZJ0urVq7V//359/PHHql27ttq0aaNXXnlFb7/9ti5duiRJmjNnjipWrKipU6eqevXqGjRokJ5++mlNnz49T3GSSAIAADhQZmam0tLS7LbMzMy/PSYiIkLt2rVTixYt7Nq3b9+uy5cv27Xfd999KleunGJjYyVJsbGxqlmzpoKCgmx9wsLClJaWpn379tn6XH/usLAw2zlyi0QSAADAgXMko6Ki5Ovra7dFRUXdNJRPP/1UO3bsuGGfpKQkubu7y8/Pz649KChISUlJtj5/TSKv7b+27+/6pKWlKSMjI9dfG8v/AAAAONCYMWMUGRlp12a1Wm/Y9+jRoxo6dKiio6Pl4eFxJ8L7R6hIAgAAOHCOpNVqlY+Pj912s0Ry+/btSklJUd26dVW4cGEVLlxY69ev15tvvqnChQsrKChIly5dUmpqqt1xycnJCg4OliQFBwfneIr72udb9fHx8ZGnp2euvzYSSQAAABd52KZ58+bas2eP4uLibFv9+vXVrVs3289FihTRmjVrbMfEx8crMTFRoaGhkqTQ0FDt2bNHKSkptj7R0dHy8fFRSEiIrc9fz3Gtz7Vz5BZD2wAAAC6iWLFiqlGjhl2bl5eXihcvbmvv06ePIiMjFRAQIB8fHw0ePFihoaF6+OGHJUmtWrVSSEiInnvuOU2ZMkVJSUkaO3asIiIibJXQAQMGaNasWRo1apR69+6ttWvXasmSJVq5cmWe4iWRBAAAuIvetT19+nS5ubmpU6dOyszMVFhYmN555x3b/kKFCmnFihUaOHCgQkND5eXlpfDwcE2aNMnWp2LFilq5cqWGDx+umTNnqkyZMnr//fcVFhaWp1gshmEYt+3OXIRng5HODgGAg5z8cYqzQwDgIF7uzkvmPJ+Y7bBzZ3w90GHndjYqkgAAACYXDi/o+NYAAABgChVJAACAu2iOpCuhIgkAAABTqEgCAAAwR9IUEkkAAACGtk0h/QYAAIApVCQBAECBZ6EiaQoVSQAAAJhCRRIAABR4VCTNoSIJAAAAU6hIAgAAUJA0hYokAAAATKEiCQAACjzmSJpDIgkAAAo8EklzGNoGAACAKVQkAQBAgUdF0hwqkgAAADCFiiQAACjwqEiaQ0USAAAAplCRBAAAoCBpChVJAAAAmEJFEgAAFHjMkTSHiiQAAABMoSIJAAAKPCqS5pBIAgCAAo9E0hyGtgEAAGAKFUkAAFDgUZE0h4okAAAATKEiCQAAQEHSFCqSAAAAMIWKJAAAKPCYI2kOFUkAAACYQkUSAAAUeFQkzSGRBAAABR6JpDkMbQMAAMAUKpIAAAAUJE2hIgkAAABTqEgCAIACjzmS5lCRBAAAgCkuk0j++OOP6t69u0JDQ/Xnn39KkhYuXKiNGzc6OTIAAJDfWSwWh235mUskkl988YXCwsLk6empnTt3KjMzU5J09uxZvfbaa06ODgAAADfiEonkq6++qjlz5mju3LkqUqSIrb1Ro0basWOHEyMDAAAFARVJc1ziYZv4+Hg1btw4R7uvr69SU1PvfEAAAKBAye8Jn6O4REUyODhYBw8ezNG+ceNGVapUyQkRAQAA4FZcIpHs16+fhg4dqi1btshisejYsWNatGiRRowYoYEDBzo7PAAAkN9ZHLjlYy4xtP3CCy8oOztbzZs314ULF9S4cWNZrVaNGDFCgwcPdnZ4AAAAuAGXSCQtFoteeukljRw5UgcPHlR6erpCQkLk7e3t7NAAAEABwBxJc1xiaPvjjz/WhQsX5O7urpCQED300EMkkQAAAC7OJRLJ4cOHKzAwUF27dtW3336rrKwsZ4cEAAAKEJb/McclEsnjx4/r008/lcVi0TPPPKNSpUopIiJCmzZtcnZoAAAAuAmXSCQLFy6s9u3ba9GiRUpJSdH06dN1+PBhNWvWTJUrV3Z2eAAAIJ9zlYrk7NmzVatWLfn4+MjHx0ehoaH67rvvbPubNm2a4/wDBgywO0diYqLatWunokWLKjAwUCNHjtSVK1fs+sTExKhu3bqyWq2qUqWK5s+fb+p7c4mHbf6qaNGiCgsL05kzZ3TkyBH98ssvzg4JAADkdy4yAl2mTBm9/vrrqlq1qgzD0IIFC9ShQwft3LlT999/v6SryyZOmjTJdkzRokVtP2dlZaldu3YKDg7Wpk2bdPz4cfXo0UNFihSxvXY6ISFB7dq104ABA7Ro0SKtWbNGffv2ValSpRQWFpaneF0mkbxw4YKWLVtmu6GyZcuqS5cuWrp0qbNDAwAAuCMef/xxu8+TJ0/W7NmztXnzZlsiWbRoUQUHB9/w+NWrV2v//v364YcfFBQUpNq1a+uVV17R6NGjNWHCBLm7u2vOnDmqWLGipk6dKkmqXr26Nm7cqOnTp+c5kXSJoe3OnTsrMDBQw4cPV6VKlRQTE6ODBw/qlVde0X333efs8AAAQD7nyKHtzMxMpaWl2W2ZmZm3jCkrK0uffvqpzp8/r9DQUFv7okWLVKJECdWoUUNjxozRhQsXbPtiY2NVs2ZNBQUF2drCwsKUlpamffv22fq0aNHC7lphYWGKjY3N8/fmEhXJQoUKacmSJQoLC1OhQoWcHQ4AAMBtExUVpYkTJ9q1jR8/XhMmTLhh/z179ig0NFQXL16Ut7e3li1bppCQEElS165dVb58eZUuXVq7d+/W6NGjFR8fry+//FKSlJSUZJdESrJ9TkpK+ts+aWlpysjIkKenZ67vzSUSyUWLFjk7BAAAUIA5cpmeMWPGKDIy0q7NarXetH+1atUUFxens2fPaunSpQoPD9f69esVEhKi/v372/rVrFlTpUqVUvPmzXXo0CGnPKDstETyzTffVP/+/eXh4aE333zzb/sOGTLkDkUFAABwe1mt1r9NHK/n7u6uKlWqSJLq1aunbdu2aebMmXr33Xdz9G3QoIEk6eDBg6pcubKCg4O1detWuz7JycmSZJtXGRwcbGv7ax8fH588VSMlJyaS06dPV7du3eTh4aHp06fftJ/FYiGRLEBG9GimVyLaatanP2rk9K/l7+Opcf1aqXmDe1U2yF8nU9P1zfp9mvjuKqWdv2h3bPd29TWkS2NVLVdCaecz9eXa3Rr+xjJJ0qN1K2lwl8aqH1JWPl4eOnj0pGZ8HKNPV+10xm0CBVa7sMd0/NixHO3/erar/jN4iOa8/ZY2x/6kpOPH5e8foKaPNdfAQUNVrFgxW98pUa8qbucOHTp4QBUrVdanS5ffwTtAfuXKC4dnZ2ffdE5lXFycJKlUqVKSpNDQUE2ePFkpKSkKDAyUJEVHR8vHx8c2PB4aGqpvv/3W7jzR0dF28zBzy2mJZEJCwg1/RsFVr3oZ9XnyYe0+8H//yJQq4aNSJX015s0V+iUhReWC/fTWC51UqqSPuo5ZaOs3pEtjDe3aWC++tUJb9x2Vl6e7ypfyt+1/uGYF7T14XNM+Wqfk0+lq+0h1vT++s86mX9R3P7HEFHCnfPzJUmVl/9/byw4dOKCB/XurZViYTqSk6MSJFA17fpQqVa6i48eO6bVXxuvEiRS9Mc1+5KrDk520d89uHfgt/k7fAuBQY8aMUZs2bVSuXDmdO3dOixcvVkxMjFatWqVDhw5p8eLFatu2rYoXL67du3dr+PDhaty4sWrVqiVJatWqlUJCQvTcc89pypQpSkpK0tixYxUREWGrig4YMECzZs3SqFGj1Lt3b61du1ZLlizRypUr8xyvS8yRnDRpkkaMGGG3DpIkZWRk6I033tDLL7/spMhwp3h5umvepK76z2tL9UKv5rb2/b8nq8sLH9k+J/x5ShNmf68PJ3ZRoUJuysrKll8xT40fEKZOz89TzM8HbX33Hjxu+/mNBWvtrvf2ZxvVvMG96tCsBokkcAf5BwTYfZ73wVyVKVtO9eo/JIvFov9Nf8u2r2zZcooYPFxjx1xdTLlw4av/ZI0aM1aSdObMaRJJ3DauUpFMSUlRjx49dPz4cfn6+qpWrVpatWqVWrZsqaNHj+qHH37QjBkzdP78eZUtW1adOnXS2LFjbccXKlRIK1as0MCBAxUaGiovLy+Fh4fbrTtZsWJFrVy5UsOHD9fMmTNVpkwZvf/++3le+kdykURy4sSJGjBgQI5E8sKFC5o4cSKJZAEwY+ST+v6nX7Ru2wG7RPJGfLw9lHb+orKysiVJzR+qKjeLRaVL+mrnpyNUzMuqzbuP6IWZ3+iPlLM3PY+vt4fiD6fc1vsAkHuXL1/Sdyu+VrcePW/6j3h6+jl5eXvbkkjAYVwjj9QHH3xw031ly5bV+vXrb3mO8uXL5xi6vl7Tpk21c+c/n97lEutIGoZxw79Edu3apYDr/uv1ejdam8nIvvK3x8C1/KvlA6pd7R6Ne+e7W/Yt7ltUY3q30IfLt9jaKt5TXG5uFo3q+ZhGTv9aXccslL9PUa14q7+KFL7xclKdmtdSvepl9dE3227bfQDIm3Vr1ujcuXN6osOTN9x/5swZzX13tp56+pk7HBmA3HJqIunv76+AgABZLBbde++9CggIsG2+vr5q2bKlnnnm7/8CiYqKkq+vr9125diWvz0GrqNMoK/eiOygXuM/Uealv/8PgGJeVi2b1ke/JCTr1bmrbe0Wi0XuRQrr+Wlf6Yctv2nr3kSFj1ukKmVLqEm9nEshNK5XWe+Oe1b/eW2pfklIzrEfwJ2xfNlSNXzkUZUMDMqxLz09XUMj/q1KlSrr3wMHOSE6FDSu8q7tu41TxwpmzJghwzDUu3dvTZw4Ub6+vrZ97u7uqlChwi2fILrR2kyBzcc7JF7cfnXuK6OggGKKXTDU1la4cCE9UqeiBjzdUL6PjlF2tiHvolZ9PaOvzl3I1LOjF+jK/x/WlqSkU2mSpF//khSeTD2vk2fPq2ywn931HqlTSV/8r5dGzfhai7/b7tibA3BTx479qa2bY+3mRF5z/ny6Bg3oq6JFvTR15iwVKVLECRECyA2nJpLh4eGSrk76bNiwoam/LG60NpPFjbk0d4t1Px9UvS7/s2t7b9yzij+SoqkfrVN2tqFiXlZ9M7OfMi9d0dMj5uWoXMbuOixJqlqupP78/3Mi/X08VcLXS4lJqbZ+j9atpC+n9tbYt7+1GxoHcOd9vfxLBQQU1yONm9i1p6enK+LffeTu7q7pb72Tp7X3gH8iv1cOHcVpGVdaWpp8fHwkSXXq1FFGRoYyMjJu2PdaP+Q/6Rcytf93++Hl8xmXdPrsBe3/PVnFvKxa8WY/eVrd1Wv8J/Lx8pCPl4ck6URqurKzDR08elLfrN+r/0V20KCopUo7f1GT/tNW8UdStP7/P8XduF5lfTm1t97+7EctX7tHQQFX16S7dOWKzqTd+P93ABwjOztbXy9fpvZPdLR7iCY9PV3/+XcfXczI0Kuvv6Hz59N1/ny6JMnfP8D2Ct3ExCPKuHBBp06eVGbmRcX/enXlhUqVK6tIEfc7f0NAAea0RNLf31/Hjx9XYGCg/Pz8bvhfAtcewsnKyrrBGVAQ1K52jx6qUV6StP/LF+z2Vev4mhKPn5Ek9Zn4qaYMe0JfTuutbMPQxh2/q8PQ921D4N3b1peXp7tG9WyuUT3/76nwDdsPKew/c+7Q3QCQpC2bNynp+DF1ePIpu/Zff9mnvbt3SZI6tG1lt2/F9z+o9D1lJEmvjB+r7T//34NyXf71ZI4+QF5RkDTHYhiG4YwLr1+/Xo0aNVLhwoVv+Sh7kyZN/nb/9TwbjPwnoQFwYSd/nOLsEAA4iJe787K5KiNuvXKIWQf/18Zh53Y2p1Uk/5oc5jVRBAAAuJ2YI2mOS6wj+f3332vjxo22z2+//bZq166trl276syZM06MDAAAFAQWi+O2/MwlEsmRI0cqLe3qEi579uxRZGSk2rZtq4SEhBxL+wAAAMA1uMQ6OQkJCQoJCZEkffHFF3r88cf12muvaceOHWrbtq2TowMAAPkdQ9vmuERF0t3dXRcuXJAk/fDDD2rV6urTegEBAbZKJQAAAFyLS1QkH3nkEUVGRqpRo0baunWrPvvsM0nSb7/9pjJlWMoBAAA4FgVJc1yiIjlr1iwVLlxYS5cu1ezZs3XPPfdIkr777ju1bt3aydEBAADgRlyiIlmuXDmtWLEiR/v06dOdEA0AACho3NwoSZrhEomkJGVlZWn58uX65Zerr7q6//779cQTT9heiQUAAADX4hKJ5MGDB9W2bVv9+eefqlatmiQpKipKZcuW1cqVK1W5cmUnRwgAAPIz5kia4xJzJIcMGaLKlSvr6NGj2rFjh3bs2KHExERVrFhRQ4YMcXZ4AAAgn7NYLA7b8jOXqEiuX79emzdvVkBAgK2tePHiev3119WoUSMnRgYAAICbcYlE0mq16ty5czna09PT5e7u7oSIAABAQZLPC4cO4xJD2+3bt1f//v21ZcsWGYYhwzC0efNmDRgwQE888YSzwwMAAMANuEQi+eabb6py5coKDQ2Vh4eHPDw81LBhQ1WpUkUzZ850dngAACCfY46kOS4xtO3n56evvvpKBw8e1P79+yVJISEhqlKlipMjAwAAwM24RCIpSR988IGmT5+uAwcOSJKqVq2qYcOGqW/fvk6ODAAA5Hf5vXLoKC6RSL788suaNm2aBg8erNDQUElSbGyshg8frsTERE2aNMnJEQIAAOB6LpFIzp49W3PnzlWXLl1sbU888YRq1aqlwYMHk0gCAACHoiBpjkskkpcvX1b9+vVztNerV09XrlxxQkQAAKAgYWjbHJd4avu5557T7Nmzc7S/99576tatmxMiAgAAwK24REVSuvqwzerVq/Xwww9LkrZs2aLExET16NFDkZGRtn7Tpk1zVogAACCfoiBpjkskknv37lXdunUlSYcOHZIklShRQiVKlNDevXtt/Sg7AwAAuA6XSCTXrVvn7BAAAEABRrHKHJeYIwkAAIC7j0tUJAEAAJyJgqQ5VCQBAABgChVJAABQ4DFH0hwqkgAAADCFiiQAACjwKEiaQyIJAAAKPIa2zWFoGwAAAKZQkQQAAAUeBUlzqEgCAADAFCqSAACgwGOOpDlUJAEAAGAKFUkAAFDgUZA0h4okAAAATKEiCQAACjzmSJpDIgkAAAo88khzGNoGAACAKVQkAQBAgcfQtjlUJAEAAGAKFUkAAFDgUZE0h4okAAAATCGRBAAABZ7F4rgtL2bPnq1atWrJx8dHPj4+Cg0N1XfffWfbf/HiRUVERKh48eLy9vZWp06dlJycbHeOxMREtWvXTkWLFlVgYKBGjhypK1eu2PWJiYlR3bp1ZbVaVaVKFc2fP9/U90YiCQAA4CLKlCmj119/Xdu3b9fPP/+sxx57TB06dNC+ffskScOHD9c333yjzz//XOvXr9exY8f01FNP2Y7PyspSu3btdOnSJW3atEkLFizQ/Pnz9fLLL9v6JCQkqF27dmrWrJni4uI0bNgw9e3bV6tWrcpzvBbDMIx/ftuuxbPBSGeHAMBBTv44xdkhAHAQL3fnzVNsOmOTw869amA9ZWZm2rVZrVZZrdZcHR8QEKA33nhDTz/9tEqWLKnFixfr6aefliT9+uuvql69umJjY/Xwww/ru+++U/v27XXs2DEFBQVJkubMmaPRo0frxIkTcnd31+jRo7Vy5Urt3bvXdo3OnTsrNTVV33//fZ7ujYokAAAo8Bw5tB0VFSVfX1+7LSoq6pYxZWVl6dNPP9X58+cVGhqq7du36/Lly2rRooWtz3333ady5copNjZWkhQbG6uaNWvakkhJCgsLU1pamq2qGRsba3eOa32unSMveGobAADAgcaMGaPIyEi7tr+rRu7Zs0ehoaG6ePGivL29tWzZMoWEhCguLk7u7u7y8/Oz6x8UFKSkpCRJUlJSkl0SeW3/tX1/1yctLU0ZGRny9PTM9b2RSAIAgALPkcv/5GUYW5KqVaumuLg4nT17VkuXLlV4eLjWr1/vsPj+CRJJAAAAF+Lu7q4qVapIkurVq6dt27Zp5syZevbZZ3Xp0iWlpqbaVSWTk5MVHBwsSQoODtbWrVvtznftqe6/9rn+Se/k5GT5+PjkqRopMUcSAADAZZb/uZHs7GxlZmaqXr16KlKkiNasWWPbFx8fr8TERIWGhkqSQkNDtWfPHqWkpNj6REdHy8fHRyEhIbY+fz3HtT7XzpEXVCQBAABcxJgxY9SmTRuVK1dO586d0+LFixUTE6NVq1bJ19dXffr0UWRkpAICAuTj46PBgwcrNDRUDz/8sCSpVatWCgkJ0XPPPacpU6YoKSlJY8eOVUREhG14fcCAAZo1a5ZGjRql3r17a+3atVqyZIlWrlyZ53hJJAEAQIHn5iKvSExJSVGPHj10/Phx+fr6qlatWlq1apVatmwpSZo+fbrc3NzUqVMnZWZmKiwsTO+8847t+EKFCmnFihUaOHCgQkND5eXlpfDwcE2aNMnWp2LFilq5cqWGDx+umTNnqkyZMnr//fcVFhaW53hZRxLAXYV1JIH8y5nrSLactdlh544e9LDDzu1sVCQBAECB5yIFybsOiSQAACjwHLn8T37GU9sAAAAwhYokAAAo8NwoSJpCRRIAAACmUJEEAAAFHnMkzaEiCQAAAFOoSAIAgAKPgqQ5VCQBAABgChVJAABQ4FlESdIMEkkAAFDgsfyPOQxtAwAAwBQqkgAAoMBj+R9zqEgCAADAFCqSAACgwKMgaQ4VSQAAAJhCRRIAABR4bpQkTclzRXLBggVauXKl7fOoUaPk5+enhg0b6siRI7c1OAAAALiuPCeSr732mjw9PSVJsbGxevvttzVlyhSVKFFCw4cPv+0BAgAAOJrF4rgtP8vz0PbRo0dVpUoVSdLy5cvVqVMn9e/fX40aNVLTpk1vd3wAAAAOx/I/5uS5Iunt7a1Tp05JklavXq2WLVtKkjw8PJSRkXF7owMAAIDLynNFsmXLlurbt6/q1Kmj3377TW3btpUk7du3TxUqVLjd8QEAADgcBUlz8lyRfPvttxUaGqoTJ07oiy++UPHixSVJ27dvV5cuXW57gAAAAHBNea5I+vn5adasWTnaJ06ceFsCAgAAuNNY/secXCWSu3fvzvUJa9WqZToYAAAA3D1ylUjWrl1bFotFhmHccP+1fRaLRVlZWbc1QAAAAEejHmlOrhLJhIQER8cBAACAu0yuEsny5cs7Og4AAACnYR1Jc/L81LYkLVy4UI0aNVLp0qVtr0WcMWOGvvrqq9saHAAAwJ3gZnHclp/lOZGcPXu2IiMj1bZtW6WmptrmRPr5+WnGjBm3Oz4AAAC4qDwnkm+99Zbmzp2rl156SYUKFbK1169fX3v27LmtwQEAANwJFovFYVt+ludEMiEhQXXq1MnRbrVadf78+dsSFAAAAFxfnhPJihUrKi4uLkf7999/r+rVq9+OmAAAAO4oi8VxW36W5zfbREZGKiIiQhcvXpRhGNq6das++eQTRUVF6f3333dEjAAAAHBBeU4k+/btK09PT40dO1YXLlxQ165dVbp0ac2cOVOdO3d2RIwAAAAOld/nMjpKnhNJSerWrZu6deumCxcuKD09XYGBgbc7LgAAALg4U4mkJKWkpCg+Pl7S1Sy+ZMmSty0oAACAOym/r/foKHl+2ObcuXN67rnnVLp0aTVp0kRNmjRR6dKl1b17d509e9YRMQIAADgUy/+Yk+dEsm/fvtqyZYtWrlyp1NRUpaamasWKFfr555/173//2xExAgAAwAXleWh7xYoVWrVqlR555BFbW1hYmObOnavWrVvf1uAAAADuhPxdN3ScPFckixcvLl9f3xztvr6+8vf3vy1BAQAAwPXlOZEcO3asIiMjlZSUZGtLSkrSyJEjNW7cuNsaHAAAwJ3gZrE4bMvPcjW0XadOHbvJogcOHFC5cuVUrlw5SVJiYqKsVqtOnDjBPEkAAIACIleJZMeOHR0cBgAAgPPk88Khw+QqkRw/fryj4wAAAMBdxvSC5AAAAPlFfl/v0VHynEhmZWVp+vTpWrJkiRITE3Xp0iW7/adPn75twQEAAMB15fmp7YkTJ2ratGl69tlndfbsWUVGRuqpp56Sm5ubJkyY4IAQAQAAHMticdyWn+U5kVy0aJHmzp2r559/XoULF1aXLl30/vvv6+WXX9bmzZsdESMAAIBDsfyPOXlOJJOSklSzZk1Jkre3t+392u3bt9fKlStvb3QAAABwWXlOJMuUKaPjx49LkipXrqzVq1dLkrZt2yar1Xp7owMAALgDGNo2J8+J5JNPPqk1a9ZIkgYPHqxx48apatWq6tGjh3r37n3bAwQAACgooqKi9OCDD6pYsWIKDAxUx44dFR8fb9enadOmslgsdtuAAQPs+iQmJqpdu3YqWrSoAgMDNXLkSF25csWuT0xMjOrWrSur1aoqVapo/vz5eY43z09tv/7667afn332WZUvX16bNm1S1apV9fjjj+c5AAAAAGdzleV/1q9fr4iICD344IO6cuWKXnzxRbVq1Ur79++Xl5eXrV+/fv00adIk2+eiRYvafs7KylK7du0UHBysTZs26fjx4+rRo4eKFCmi1157TZKUkJCgdu3aacCAAVq0aJHWrFmjvn37qlSpUgoLC8t1vBbDMIzbcN9KSUnR+++/rxdffPF2nO4f8Www0tkhAHCQkz9OcXYIABzEy915yVzEsl8cdu63n6xu+tgTJ04oMDBQ69evV+PGjSVdrUjWrl1bM2bMuOEx3333ndq3b69jx44pKChIkjRnzhyNHj1aJ06ckLu7u0aPHq2VK1dq7969tuM6d+6s1NRUff/997mO77YtSH78+HGNGzfOJRLJMz+94ewQADiI/4ODnB0CAAfJ2DnLadfO81y/PMjMzFRmZqZdm9VqzdWzJdceag4ICLBrX7RokT7++GMFBwfr8ccf17hx42xVydjYWNWsWdOWREpSWFiYBg4cqH379qlOnTqKjY1VixYt7M4ZFhamYcOG5eneHPm9AQAAFHhRUVHy9fW126Kiom55XHZ2toYNG6ZGjRqpRo0atvauXbvq448/1rp16zRmzBgtXLhQ3bt3t+1PSkqySyIl2T4nJSX9bZ+0tDRlZGTk+t54RSIAACjwHDlHcsyYMYqMjLRry001MiIiQnv37tXGjRvt2vv372/7uWbNmipVqpSaN2+uQ4cOqXLlyrcn6FwikQQAAAWemwOnZ+Z2GPuvBg0apBUrVmjDhg0qU6bM3/Zt0KCBJOngwYOqXLmygoODtXXrVrs+ycnJkqTg4GDb/15r+2sfHx8feXp65jrOXCeS12fS1ztx4kSuLwoAAICcDMPQ4MGDtWzZMsXExKhixYq3PCYuLk6SVKpUKUlSaGioJk+erJSUFAUGBkqSoqOj5ePjo5CQEFufb7/91u480dHRCg0NzVO8uU4kd+7cecs+154mAgAAuJs4siKZFxEREVq8eLG++uorFStWzDan0dfXV56enjp06JAWL16stm3bqnjx4tq9e7eGDx+uxo0bq1atWpKkVq1aKSQkRM8995ymTJmipKQkjR07VhEREbbK6IABAzRr1iyNGjVKvXv31tq1a7VkyZI8v6Xwti3/40ouXrl1HwB3J57aBvIvZz61Hfn1rw4797Qn7st135vN1Zw3b5569uypo0ePqnv37tq7d6/Onz+vsmXL6sknn9TYsWPl4+Nj63/kyBENHDhQMTEx8vLyUnh4uF5//XUVLvx/NcSYmBgNHz5c+/fvV5kyZTRu3Dj17NkzT/dGIgngrkIiCeRfzkwkn/8m/tadTJr6eDWHndvZWP4HAAAApvDUNgAAKPBcZY7k3YaKJAAAAEyhIgkAAAo8B65Hnq+Zqkj++OOP6t69u0JDQ/Xnn39KkhYuXJhj5XUAAIC7gZvF4rAtP8tzIvnFF18oLCxMnp6e2rlzp+0l5GfPntVrr7122wMEAACAa8pzIvnqq69qzpw5mjt3rooUKWJrb9SokXbs2HFbgwMAALgT3By45Wd5vr/4+PgbvsHG19dXqamptyMmAAAA3AXynEgGBwfr4MGDOdo3btyoSpUq3ZagAAAA7iSLxXFbfpbnRLJfv34aOnSotmzZIovFomPHjmnRokUaMWKEBg4c6IgYAQAA4ILyvPzPCy+8oOzsbDVv3lwXLlxQ48aNZbVaNWLECA0ePNgRMQIAADhUfn+62lHynEhaLBa99NJLGjlypA4ePKj09HSFhITI29vbEfEBAADARZlekNzd3V0hISG3MxYAAACnoCBpTp4TyWbNmsnyN9/22rVr/1FAAAAAdxrv2jYnz4lk7dq17T5fvnxZcXFx2rt3r8LDw29XXAAAAHBxeU4kp0+ffsP2CRMmKD09/R8HBAAAcKfxsI05t23B9e7du+vDDz+8XacDAACAizP9sM31YmNj5eHhcbtOBwAAcMdQkDQnz4nkU089ZffZMAwdP35cP//8s8aNG3fbAgMAAIBry3Mi6evra/fZzc1N1apV06RJk9SqVavbFhgAAMCdwlPb5uQpkczKylKvXr1Us2ZN+fv7OyomAAAA3AXy9LBNoUKF1KpVK6WmpjooHAAAgDvP4sA/+Vmen9quUaOGfv/9d0fEAgAA4BRuFsdt+VmeE8lXX31VI0aM0IoVK3T8+HGlpaXZbQAAACgYcj1HctKkSXr++efVtm1bSdITTzxh96pEwzBksViUlZV1+6MEAABwoPxeOXSUXCeSEydO1IABA7Ru3TpHxgMAAIC7RK4TScMwJElNmjRxWDAAAADOYGFFclPyNEeSLxkAAADX5GkdyXvvvfeWyeTp06f/UUAAAAB3GnMkzclTIjlx4sQcb7YBAABAwZSnRLJz584KDAx0VCwAAABOwew9c3KdSDI/EgAA5Fdu5Dmm5Pphm2tPbQMAAABSHiqS2dnZjowDAADAaXjYxpw8vyIRAAAAkPL4sA0AAEB+xBRJc6hIAgAAwBQqkgAAoMBzEyVJM6hIAgAAwBQqkgAAoMBjjqQ5JJIAAKDAY/kfcxjaBgAAgClUJAEAQIHHKxLNoSIJAAAAU6hIAgCAAo+CpDlUJAEAAGAKFUkAAFDgMUfSHCqSAAAAMIWKJAAAKPAoSJpDIgkAAAo8hmjN4XsDAACAKSSSAACgwLNYLA7b8iIqKkoPPvigihUrpsDAQHXs2FHx8fF2fS5evKiIiAgVL15c3t7e6tSpk5KTk+36JCYmql27dipatKgCAwM1cuRIXblyxa5PTEyM6tatK6vVqipVqmj+/Pl5/t5IJAEAAFzE+vXrFRERoc2bNys6OlqXL19Wq1atdP78eVuf4cOH65tvvtHnn3+u9evX69ixY3rqqads+7OystSuXTtdunRJmzZt0oIFCzR//ny9/PLLtj4JCQlq166dmjVrpri4OA0bNkx9+/bVqlWr8hSvxTAM45/ftmu5eOXWfQDcnfwfHOTsEAA4SMbOWU679kc/H3XYuXvUL2v62BMnTigwMFDr169X48aNdfbsWZUsWVKLFy/W008/LUn69ddfVb16dcXGxurhhx/Wd999p/bt2+vYsWMKCgqSJM2ZM0ejR4/WiRMn5O7urtGjR2vlypXau3ev7VqdO3dWamqqvv/++1zHR0USAADAgTIzM5WWlma3ZWZm5urYs2fPSpICAgIkSdu3b9fly5fVokULW5/77rtP5cqVU2xsrCQpNjZWNWvWtCWRkhQWFqa0tDTt27fP1uev57jW59o5cotEEgAAFHhuFovDtqioKPn6+tptUVFRt4wpOztbw4YNU6NGjVSjRg1JUlJSktzd3eXn52fXNygoSElJSbY+f00ir+2/tu/v+qSlpSkjIyPX3xvL/wAAADjQmDFjFBkZaddmtVpveVxERIT27t2rjRs3Oiq0f4xEEgAAFHiOXI/carXmKnH8q0GDBmnFihXasGGDypQpY2sPDg7WpUuXlJqaaleVTE5OVnBwsK3P1q1b7c537anuv/a5/knv5ORk+fj4yNPTM9dxMrQNAAAKPIvFcVteGIahQYMGadmyZVq7dq0qVqxot79evXoqUqSI1qxZY2uLj49XYmKiQkNDJUmhoaHas2ePUlJSbH2io6Pl4+OjkJAQW5+/nuNan2vnyC0qkgAAAC4iIiJCixcv1ldffaVixYrZ5jT6+vrK09NTvr6+6tOnjyIjIxUQECAfHx8NHjxYoaGhevjhhyVJrVq1UkhIiJ577jlNmTJFSUlJGjt2rCIiImyV0QEDBmjWrFkaNWqUevfurbVr12rJkiVauXJlnuJl+R8AdxWW/wHyL2cu//PJzj8ddu4ude7Jdd+bLWA+b9489ezZU9LVBcmff/55ffLJJ8rMzFRYWJjeeecd27C1JB05ckQDBw5UTEyMvLy8FB4ertdff12FC/9fDTEmJkbDhw/X/v37VaZMGY0bN852jVzHSyIJ4G5CIgnkXySSdx+GtgEAQIHHQyPm8L0BAADAFCqSAACgwLvZ3ET8PSqSAAAAMIWKJAAAKPCoR5pDRRIAAACmUJEEAAAFHnMkzSGRBAAABR5DtObwvQEAAMAUKpIAAKDAY2jbHCqSAAAAMIWKJAAAKPCoR5pDRRIAAACmUJEEAAAFHlMkzaEiCQAAAFOoSAIAgALPjVmSppBIAgCAAo+hbXMY2gYAAIApLpVIXrp0SfHx8bpy5YqzQwEAAAWIxYF/8jOXSCQvXLigPn36qGjRorr//vuVmJgoSRo8eLBef/11J0cHAACAG3GJRHLMmDHatWuXYmJi5OHhYWtv0aKFPvvsMydGBgAACgKLxXFbfuYSD9ssX75cn332mR5++GG7d13ef//9OnTokBMjAwAAwM24RCJ54sQJBQYG5mg/f/48L1EHAAAOx/I/5rjE0Hb9+vW1cuVK2+dryeP777+v0NBQZ4UFAACAv+ESFcnXXntNbdq00f79+3XlyhXNnDlT+/fv16ZNm7R+/XpnhwcAAPI5BkDNcYmK5COPPKK4uDhduXJFNWvW1OrVqxUYGKjY2FjVq1fP2eEBAIB8jodtzHGJiqQkVa5cWXPnznV2GAAAAMgll6hItmjRQvPnz1daWpqzQwEAAAUQC5Kb4xKJ5P33368xY8YoODhY//rXv/TVV1/p8uXLzg4LAAAAf8MlEsmZM2fqzz//1PLly+Xl5aUePXooKChI/fv352EbAADgcG4Wx235mUskkpLk5uamVq1aaf78+UpOTta7776rrVu36rHHHnN2aAAAALgBl3nY5pqkpCR9+umn+vjjj7V792499NBDzg4JAADkc/l9LqOjuERFMi0tTfPmzVPLli1VtmxZzZ49W0888YQOHDigzZs3Ozs8AAAA3IBLVCSDgoLk7++vZ599VlFRUapfv76zQwIAAAVIfl/v0VFcIpH8+uuv1bx5c7m5uUSBFAAAFDAMbZvjEolky5YtnR0CAAAA8shpiWTdunW1Zs0a+fv7q06dOrL8TU15x44ddzAyAABQ0OT3ZXocxWmJZIcOHWS1Wm0//10iCQAAANdjMQzDcHYQt9vFK86OAICj+D84yNkhAHCQjJ2znHbtH38747BzP3qvv8PO7Wwu8XRLpUqVdOrUqRztqampqlSpkhMiAgAAwK24xMM2hw8fVlZWVo72zMxM/fHHH06ICM6WnJysGdPe0E8//qiLFzNUtlx5TXr1Nd1fo6YkadyLL+jrr5bZHdOw0SOa/d4HkqRtW7eob68eNzz3ok8/V42atRx7AwByGNGrpV4Z0kGzFq3TyP99IUl666XOeqxBNZUq6av0jExt3pWgsTO/0m+Hk+2O7f54Aw3p/piqlg9U2vmL+jJ6p4a/vsS2v0bV0prxwjOqd395nTyTrtmfrte0BT/c0fvD3Y0ZduY4NZH8+uuvbT+vWrVKvr6+ts9ZWVlas2aNKlas6IzQ4ERpZ8+qZ/cuqv9QA709Z678A/yVeOSIfHx87fo1euRRTXo1yvbZ3d3d9nPt2nW0JmajXf+335qpLVtibckogDunXkg59enUSLt/sy8O7PzlqD79bpuOHj+jAN+iemlAO614J0L3tR+v7OyrM6+GdH9MQ597TC9OX66tew/Ly9Nd5UsXt52jmJeHvnlnkNZt+VWDJ3+qGlXv0Zzx3ZR6LkMffvnTHb1PoKBxaiLZsWNHSZLFYlF4eLjdviJFiqhChQqaOnWqEyKDM334wVwFBQfrlcn/lySWKVM2Rz93d3eVKFnyhucoct2+y5cva926NerStTsPdgF3mJenu+a91lP/eeUTvdC3td2+vyZ6icdPa+Lb32jbkhdVvnRxJfxxUn7FPDX+P+3VadgcxWz9zdZ374Fjtp87t60v9yKF9O8Ji3T5SpZ++T1JtardoyHdm5FIItf4l8Ecp86RzM7OVnZ2tsqVK6eUlBTb5+zsbGVmZio+Pl7t27d3ZohwgvXr1ur++2toxPAhavpoqJ7p1FFffL4kR7+ft21V00dD9US7ML06abxSU28+UXr9urU6m5qqjk92cmToAG5gxphn9f2Pe7VuS/zf9ivq4a4eTzyshD9O6o+kq7/PzR++T25uFpUO9NPOL8bq4Pev6OP/9laZID/bcQ1qVdRPOw7q8pX/myIVvekXVasYLL9ing65J+Q/bhaLw7b8zCXmSCYkJJg+NjMzU5mZmXZtRiGrbWkh3H3++OOolnz2iZ4L76U+/Qdo3549+m/UqypSpIie6PikJKnhI4+qeYuWuqdMGR09elRvzZim//y7nxYu/kyFChXKcc5lXy5Vw0aPKCg4+E7fDlCg/SusnmrfV1aPdJ9y0z79//WoJg/rKO+iVsUnJKndwFm2pLBimRJyc7NoVO9WGvHGF0pLz9D4iPZaMXuQHnwmSpevZCmouI8O/2n/wGbK6XOSpKASPko9l+G4GwQKOJdIJCXp/PnzWr9+vRITE3Xp0iW7fUOGDLnpcVFRUZo4caJd20vjxmvsyxMcESbugOxsQ/fXqKEhwyIlSdWrh+jgwQP6fMmntkSyTdt2tv5V762me++tpnatW+jnbVvV4OFQu/MlJyVp008b9cbUGXfsHgBIZYL89MbITmo/cJYyL918XbZPv9umNVt+VXAJHw3r0UIf/7e3Hus1TZmXrshisci9SGE9P2Wp1mz+VZIUPma+Dke/piYP3qsfYn+5U7eDfC5/1w0dxyUSyZ07d6pt27a6cOGCzp8/r4CAAJ08eVJFixZVYGDg3yaSY8aMUWRkpF2bUYhq5N2sZMmSqlS5sl1bpUqV9EP0qpseU6ZsWfn7+ysx8UiORHL5si/k6+enJs0ec0i8AG6sTvVyCiruo9jFo21thQsX0iN1K2vAs43l22CYsrMNpaVfVFr6RR1KPKGtuw/r+IYp6vDYA1ry/XYlnUyTJP36e5LtHCfPpOtkarrKBl9dmy/5VJqCihezu3ZgwNXPyf//eACO4RKJ5PDhw/X4449rzpw58vX11ebNm1WkSBF1795dQ4cO/dtjrdacw9gsSH53q12nrg5fN93hyOHDKl36npsek5yUpNTUVJUsYf/wjWEY+mr5l3r8iY4qUqSIQ+IFcGPrtsar3tOT7drem9hd8QnJmjo/2vZU9l9ZLBZZdLUKKUmxcb9LkqpWCNSfKamSJH+foirh563E46clSVt2J2hCxOMqXNhNV65kS7o6tzI+IYlhbeQeJUlTXGJB8ri4OD3//PNyc3NToUKFlJmZqbJly2rKlCl68cUXnR0e7rDuPcK1Z/cuvf/eHCUeOaJvV3yjpUuX6NkuXSVJF86f17T//Ve7d8Xpzz//0JbNsRo6+D8qW668Gj7yqN25tm7ZrD//+ENPdXraGbcCFGjpFzK1/9Bxu+18xiWdPnte+w8dV4V7imtE71aqU72sygb76+EHKmrRG32UkXlZqzbukyQdTEzRN+t26X8jn9bDD1RUSOVSmjvpOcUfTtb6n68+xf3Zdz/r0uUszRnfTdUrBevpVnUV0bWp3vx4nTNvHygQXKIiWaRIEbm5Xc1pAwMDlZiYqOrVq8vX11dHjx51cnS402rUrKVpM2fpzRnT9O7st3VPmTIaNfpFtWv/hCTJrVAh/Rb/m77+arnOpZ1TYGCgQhs2UsTgoXZrSUrSsi+WqnbtOqpYqfKNLgXAiTIvXVGjOpU1qGtT+fsUVcqpc9q446Ca9ZyqE2fSbf36jFuoKSOe0pdvDlR2tqGN2w+oQ8TbtupjWvpFPf6fWZrxwjPatHi0TqWmK+q971j6B3lioSRpiku8a7tVq1bq2bOnunbtqn79+mn37t0aMmSIFi5cqDNnzmjLli15Oh9D20D+xbu2gfzLme/a3nLorMPO3aCy76073aVcYmj7tddeU6lSpSRJkydPlr+/vwYOHKgTJ07ovffec3J0AAAgv7NYHLflZy6RSNavX1/NmjWTdHVo+/vvv1daWpq2b9+uBx54wMnRAQCA/M7iwC2vNmzYoMcff1ylS5eWxWLR8uXL7fb37Nnz6oNpf9lat7Z/a9Tp06fVrVs3+fj4yM/PT3369FF6erpdn927d+vRRx+Vh4eH7dmUvHKJRBIAAABXnT9/Xg888IDefvvtm/Zp3bq1jh8/bts++eQTu/3dunXTvn37FB0drRUrVmjDhg3q37+/bX9aWppatWql8uXLa/v27XrjjTc0YcKEPI8Eu8TDNnXq1Lnh+48tFos8PDxUpUoV9ezZ01a1BAAAuK1caAi6TZs2atOmzd/2sVqtCr7J29p++eUXff/999q2bZvq168vSXrrrbfUtm1b/e9//1Pp0qW1aNEiXbp0SR9++KHc3d11//33Ky4uTtOmTbNLOG/FJSqSrVu31u+//y4vLy81a9ZMzZo1k7e3tw4dOqQHH3xQx48fV4sWLfTVV185O1QAAIA8yczMVFpamt12/eud8yomJkaBgYGqVq2aBg4cqFOn/u81obGxsfLz87MlkZLUokULubm52R5gjo2NVePGje1WOwkLC1N8fLzOnDmT6zhcIpE8efKknn/+ef3444+aOnWqpk6dqg0bNmjEiBE6f/68Vq9erbFjx+qVV15xdqgAACAfsjjwT1RUlHx9fe22qKgo07G2bt1aH330kdasWaP//ve/Wr9+vdq0aaOsrKvvqE9KSlJgYKDdMYULF1ZAQICSkpJsfYKCguz6XPt8rU9uuMTQ9pIlS7R9+/Yc7Z07d1a9evU0d+5cdenSRdOmTXNCdAAAAObd6HXO17+VLy86d+5s+7lmzZqqVauWKleurJiYGDVv3tz0ec1wiYqkh4eHNm3alKN906ZN8vDwkCRlZ2fbfgYAALidHLn8j9VqlY+Pj932TxLJ61WqVEklSpTQwYMHJUnBwcFKSUmx63PlyhWdPn3aNq8yODhYycnJdn2ufb7Z3MsbcYmK5ODBgzVgwABt375dDz74oCRp27Ztev/9922vSFy1apVq167txCgBAABczx9//KFTp07Z1uQODQ1Vamqqtm/frnr16kmS1q5dq+zsbDVo0MDW56WXXtLly5dVpEgRSVJ0dLSqVasmf3//XF/bJd5sI0mLFi3SrFmzFB8fL0mqVq2aBg8erK5dr75fOSMjw/YU963wZhsg/+LNNkD+5cw32+w4nOawc9et4JOn/unp6bbqYp06dTRt2jQ1a9ZMAQEBCggI0MSJE9WpUycFBwfr0KFDGjVqlM6dO6c9e/bYKp1t2rRRcnKy5syZo8uXL6tXr16qX7++Fi9eLEk6e/asqlWrplatWmn06NHau3evevfurenTp+fpqW2XSSRvJxJJIP8ikQTyL6cmkkccmEiWz1siGRMTc8MlD8PDwzV79mx17NhRO3fuVGpqqkqXLq1WrVrplVdesXt45vTp0xo0aJC++eYbubm5qVOnTnrzzTfl7e1t67N7925FRERo27ZtKlGihAYPHqzRo0fnKVaXSSRTU1O1dOlS/f777xoxYoQCAgK0Y8cOBQUF6Z577snTuUgkgfyLRBLIv0gk7z4uMUdy9+7datGihXx9fXX48GH17dtXAQEB+vLLL5WYmKiPPvrI2SECAIB8zOJKK5LfRVziqe3IyEj17NlTBw4csJsD2bZtW23YsMGJkQEAAOBmXKIiuW3bNr377rs52u+55548LYoJAABgxg3e1IxccImKpNVqVVpazrkJv/32m0qWLOmEiAAAAHArLpFIPvHEE5o0aZIuX74sSbJYLEpMTNTo0aPVqVMnJ0cHAADyO4sDt/zMJRLJqVOnKj09XYGBgcrIyFCTJk1UpUoVeXt7a/Lkyc4ODwAAADfgEnMkfX19FR0drZ9++km7du1Senq66tatqxYtWjg7NAAAUBDk99Khg7hEIilJa9as0Zo1a5SSkqLs7Gz9+uuvttXXP/zwQydHBwAA8jOW/zHHJRLJiRMnatKkSapfv75KlSolC49OAQAAuDyXSCTnzJmj+fPn67nnnnN2KAAAoACihmWOSzxsc+nSJTVs2NDZYQAAACAPXCKR7Nu3r20+JAAAwJ3G8j/muMTQ9sWLF/Xee+/phx9+UK1atVSkSBG7/dOmTXNSZAAAALgZl0gkd+/erdq1a0uS9u7da7ePB28AAIDDkW6Y4hKJ5Lp165wdAgAAAPLIJRJJAAAAZ2IdSXNc4mEbAAAA3H2oSAIAgAKPRzLMIZEEAAAFHnmkOQxtAwAAwBQqkgAAAJQkTaEiCQAAAFOoSAIAgAKP5X/MoSIJAAAAU6hIAgCAAo/lf8yhIgkAAABTqEgCAIACj4KkOSSSAAAAZJKmMLQNAAAAU6hIAgCAAo/lf8yhIgkAAABTqEgCAIACj+V/zKEiCQAAAFOoSAIAgAKPgqQ5VCQBAABgChVJAAAASpKmkEgCAIACj+V/zGFoGwAAAKZQkQQAAAUey/+YQ0USAAAAplCRBAAABR4FSXOoSAIAAMAUKpIAAACUJE2hIgkAAABTqEgCAIACj3UkzSGRBAAABR7L/5jD0DYAAABMoSIJAAAKPAqS5lCRBAAAgClUJAEAQIHHHElzqEgCAADAFBJJAAAAWRy45c2GDRv0+OOPq3Tp0rJYLFq+fLndfsMw9PLLL6tUqVLy9PRUixYtdODAAbs+p0+fVrdu3eTj4yM/Pz/16dNH6enpdn12796tRx99VB4eHipbtqymTJmS51hJJAEAAFzI+fPn9cADD+jtt9++4f4pU6bozTff1Jw5c7RlyxZ5eXkpLCxMFy9etPXp1q2b9u3bp+joaK1YsUIbNmxQ//79bfvT0tLUqlUrlS9fXtu3b9cbb7yhCRMm6L333stTrBbDMAxzt+m6Ll5xdgQAHMX/wUHODgGAg2TsnOW0a/+Zeslh577Hz930sRaLRcuWLVPHjh0lXa1Gli5dWs8//7xGjBghSTp79qyCgoI0f/58de7cWb/88otCQkK0bds21a9fX5L0/fffq23btvrjjz9UunRpzZ49Wy+99JKSkpLk7n41vhdeeEHLly/Xr7/+muv4qEgCAIACz5ED25mZmUpLS7PbMjMzTcWZkJCgpKQktWjRwtbm6+urBg0aKDY2VpIUGxsrPz8/WxIpSS1atJCbm5u2bNli69O4cWNbEilJYWFhio+P15kzZ3IdD4kkAACAA0VFRcnX19dui4qKMnWupKQkSVJQUJBde1BQkG1fUlKSAgMD7fYXLlxYAQEBdn1udI6/XiM3WP4HAAAUeI5c/mfMmDGKjIy0a7NarY674B1EIgkAAOBAVqv1tiWOwcHBkqTk5GSVKlXK1p6cnKzatWvb+qSkpNgdd+XKFZ0+fdp2fHBwsJKTk+36XPt8rU9uMLQNAAAKPIsD/9xOFStWVHBwsNasWWNrS0tL05YtWxQaGipJCg0NVWpqqrZv327rs3btWmVnZ6tBgwa2Phs2bNDly5dtfaKjo1WtWjX5+/vnOh4SSQAAABeSnp6uuLg4xcXFSbr6gE1cXJwSExNlsVg0bNgwvfrqq/r666+1Z88e9ejRQ6VLl7Y92V29enW1bt1a/fr109atW/XTTz9p0KBB6ty5s0qXLi1J6tq1q9zd3dWnTx/t27dPn332mWbOnJljCP5WGNoGAABwoVck/vzzz2rWrJnt87XkLjw8XPPnz9eoUaN0/vx59e/fX6mpqXrkkUf0/fffy8PDw3bMokWLNGjQIDVv3lxubm7q1KmT3nzzTdt+X19frV69WhEREapXr55KlCihl19+2W6tydxgHUkAdxXWkQTyL2euI5mUdvnWnUwK9inisHM7GxVJAABQ4LlQQfKuQiIJAAAKPEcu/5Of8bANAAAATKEiCQAACrzbvUxPQUFFEgAAAKZQkQQAAKAgaQoVSQAAAJhCRRIAABR4FCTNoSIJAAAAU6hIAgCAAo91JM0hkQQAAAUey/+Yw9A2AAAATKEiCQAACjyGts2hIgkAAABTSCQBAABgCokkAAAATGGOJAAAKPCYI2kOFUkAAACYQkUSAAAUeKwjaQ6JJAAAKPAY2jaHoW0AAACYQkUSAAAUeBQkzaEiCQAAAFOoSAIAAFCSNIWKJAAAAEyhIgkAAAo8lv8xh4okAAAATKEiCQAACjzWkTSHiiQAAABMoSIJAAAKPAqS5pBIAgAAkEmawtA2AAAATKEiCQAACjyW/zGHiiQAAABMoSIJAAAKPJb/MYeKJAAAAEyxGIZhODsIwKzMzExFRUVpzJgxslqtzg4HwG3E7zfg+kgkcVdLS0uTr6+vzp49Kx8fH2eHA+A24vcbcH0MbQMAAMAUEkkAAACYQiIJAAAAU0gkcVezWq0aP348E/GBfIjfb8D18bANAAAATKEiCQAAAFNIJAEAAGAKiSQAAABMIZHEXSkmJkYWi0Wpqal/269ChQqaMWPGHYkJgPNMmDBBtWvXdnYYQIHDwza4K126dEmnT59WUFCQLBaL5s+fr2HDhuVILE+cOCEvLy8VLVrUOYECuO0sFouWLVumjh072trS09OVmZmp4sWLOy8woAAq7OwAADPc3d0VHBx8y34lS5a8A9EAcDZvb295e3s7OwygwGFoGw7TtGlTDRo0SIMGDZKvr69KlCihcePG6VoR/MyZM+rRo4f8/f1VtGhRtWnTRgcOHLAdf+TIET3++OPy9/eXl5eX7r//fn377beS7Ie2Y2Ji1KtXL509e1YWi0UWi0UTJkyQZD+03bVrVz377LN2MV6+fFklSpTQRx99JEnKzs5WVFSUKlasKE9PTz3wwANaunSpg78p4O7QtGlTDRkyRKNGjVJAQICCg4Ntv2uSlJqaqr59+6pkyZLy8fHRY489pl27dtmd49VXX1VgYKCKFSumvn376oUXXrAbkt62bZtatmypEiVKyNfXV02aNNGOHTts+ytUqCBJevLJJ2WxWGyf/zq0vXr1anl4eOQYoRg6dKgee+wx2+eNGzfq0Ucflaenp8qWLashQ4bo/Pnz//h7AgoSEkk41IIFC1S4cGFt3bpVM2fO1LRp0/T+++9Lknr27Kmff/5ZX3/9tWJjY2UYhtq2bavLly9LkiIiIpSZmakNGzZoz549+u9//3vDikPDhg01Y8YM+fj46Pjx4zp+/LhGjBiRo1+3bt30zTffKD093da2atUqXbhwQU8++aQkKSoqSh999JHmzJmjffv2afjw4erevbvWr1/viK8HuOssWLBAXl5e2rJli6ZMmaJJkyYpOjpakvSvf/1LKSkp+u6777R9+3bVrVtXzZs31+nTpyVJixYt0uTJk/Xf//5X27dvV7ly5TR79my78587d07h4eHauHGjNm/erKpVq6pt27Y6d+6cpKuJpiTNmzdPx48ft33+q+bNm8vPz09ffPGFrS0rK0ufffaZunXrJkk6dOiQWrdurU6dOmn37t367LPPtHHjRg0aNOj2f2lAfmYADtKkSROjevXqRnZ2tq1t9OjRRvXq1Y3ffvvNkGT89NNPtn0nT540PD09jSVLlhiGYRg1a9Y0JkyYcMNzr1u3zpBknDlzxjAMw5g3b57h6+ubo1/58uWN6dOnG4ZhGJcvXzZKlChhfPTRR7b9Xbp0MZ599lnDMAzj4sWLRtGiRY1NmzbZnaNPnz5Gly5d8nz/QH7TpEkT45FHHrFre/DBB43Ro0cbP/74o+Hj42NcvHjRbn/lypWNd9991zAMw2jQoIERERFht79Ro0bGAw88cNNrZmVlGcWKFTO++eYbW5skY9myZXb9xo8fb3eeoUOHGo899pjt86pVqwyr1Wr7O6NPnz5G//797c7x448/Gm5ubkZGRsZN4wFgj4okHOrhhx+WxWKxfQ4NDdWBAwe0f/9+FS5cWA0aNLDtK168uKpVq6ZffvlFkjRkyBC9+uqratSokcaPH6/du3f/o1gKFy6sZ555RosWLZIknT9/Xl999ZWtQnHw4EFduHBBLVu2tM238vb21kcffaRDhw79o2sD+UWtWrXsPpcqVUopKSnatWuX0tPTVbx4cbvfn4SEBNvvT3x8vB566CG746//nJycrH79+qlq1ary9fWVj4+P0tPTlZiYmKc4u3XrppiYGB07dkzS1Wpou3bt5OfnJ0natWuX5s+fbxdrWFiYsrOzlZCQkKdrAQUZD9vAZfXt21dhYWFauXKlVq9eraioKE2dOlWDBw82fc5u3bqpSZMmSklJUXR0tDw9PdW6dWtJsg15r1y5Uvfcc4/dcbzrF7iqSJEidp8tFouys7OVnp6uUqVKKSYmJscx15K33AgPD9epU6c0c+ZMlS9fXlarVaGhobp06VKe4nzwwQdVuXJlffrppxo4cKCWLVum+fPn2/anp6fr3//+t4YMGZLj2HLlyuXpWkBBRiIJh9qyZYvd52tznkJCQnTlyhVt2bJFDRs2lCSdOnVK8fHxCgkJsfUvW7asBgwYoAEDBmjMmDGaO3fuDRNJd3d3ZWVl3TKehg0bqmzZsvrss8/03Xff6V//+pftH8aQkBBZrVYlJiaqSZMm/+S2gQKnbt26SkpKUuHChW0PwFyvWrVq2rZtm3r06GFru36O408//aR33nlHbdu2lSQdPXpUJ0+etOtTpEiRXP2+d+vWTYsWLVKZMmXk5uamdu3a2cW7f/9+ValSJbe3COAGGNqGQyUmJioyMlLx8fH65JNP9NZbb2no0KGqWrWqOnTooH79+mnjxo3atWuXunfvrnvuuUcdOnSQJA0bNkyrVq1SQkKCduzYoXXr1ql69eo3vE6FChWUnp6uNWvW6OTJk7pw4cJNY+ratavmzJmj6Oho27C2JBUrVkwjRozQ8OHDtWDBAh06dEg7duzQW2+9pQULFtzeLwbIZ1q0aKHQ0FB17NhRq1ev1uHDh7Vp0ya99NJL+vnnnyVJgwcP1gcffKAFCxbowIEDevXVV7V792676S9Vq1bVwoUL9csvv2jLli3q1q2bPD097a5VoUIFrVmzRklJSTpz5sxNY+rWrZt27NihyZMn6+mnn7YbWRg9erQ2bdqkQYMGKS4uTgcOHNBXX33FwzZAHpFIwqF69OihjIwMPfTQQ4qIiNDQoUPVv39/SVefuqxXr57at2+v0NBQGYahb7/91lYhzMrKUkREhKpXr67WrVvr3nvv1TvvvHPD6zRs2FADBgzQs88+q5IlS2rKlCk3jalbt27av3+/7rnnHjVq1Mhu3yuvvKJx48YpKirKdt2VK1eqYsWKt+kbAfIni8Wib7/9Vo0bN1avXr107733qnPnzjpy5IiCgoIkXf3dGzNmjEaMGKG6desqISFBPXv2lIeHh+08H3zwgc6cOaO6devqueee05AhQxQYGGh3ralTpyo6Olply5ZVnTp1bhpTlSpV9NBDD2n37t12/9EoXZ3ruX79ev3222969NFHVadOHb388ssqXbr0bfxWgPyPN9vAYZo2baratWvzikIAN9WyZUsFBwdr4cKFzg4FgAnMkQQA3BEXLlzQnDlzFBYWpkKFCumTTz7RDz/8YFuHEsDdh0QSAHBHXBv+njx5si5evKhq1arpiy++UIsWLZwdGgCTGNoGAACAKTxsAwAAAFNIJAEAAGAKiSQAAABMIZEEAACAKSSSAAAAMIVEEoBpPXv2VMeOHW2fmzZtqmHDht3xOGJiYmSxWJSamuqwa1x/r2bciTgB4E4ikQTymZ49e8pischiscjd3V1VqlTRpEmTdOXKFYdf+8svv9Qrr7ySq753OqmqUKECb1kCgNuMBcmBfKh169aaN2+eMjMz9e233yoiIkJFihTRmDFjcvS9dOmS3N3db8t1AwICbst5AAB3ByqSQD5ktVoVHBys8uXLa+DAgWrRooW+/vprSf83RDt58mSVLl1a1apVkyQdPXpUzzzzjPz8/BQQEKAOHTro8OHDtnNmZWUpMjJSfn5+Kl68uEaNGqXr32dw/dB2ZmamRo8erbJly8pqtapKlSr64IMPdPjwYTVr1kyS5O/vL4vFop49e0qSsrOzFRUVpYoVK8rT01MPPPCAli5danedb7/9Vvfee688PT3VrFkzuzjNyMrKUp8+fWzXrFatmmbOnHnDvhMnTlTJkiXl4+OjAQMG6NKlS7Z9uYn9r44cOaLHH39c/v7+8vLy0v33369vv/32H90LANxJVCSBAsDT01OnTp2yfV6zZo18fHxs7zi+fPmywsLCFBoaqh9//FGFCxfWq6++qtatW2v37t1yd3fX1KlTNX/+fH344YeqXr26pk6dqmXLlumxxx676XV79Oih2NhYvfnmm3rggQeUkJCgkydPqmzZsvriiy/UqVMnxcfHy8fHR56enpKkqKgoffzxx5ozZ46qVq2qDRs2qHv37ipZsqSaNGmio0eP6qmnnlJERIT69++vn3/+Wc8///w/+n6ys7NVpkwZff755ypevLg2bdqk/v37q1SpUnrmmWfsvjcPDw/FxMTo8OHD6tWrl4oXL67JkyfnKvbrRURE6NKlS9qwYYO8vLy0f/9+eXt7/6N7AYA7ygCQr4SHhxsdOnQwDMMwsrOzjejoaMNqtRojRoyw7Q8KCjIyMzNtxyxcuNCoVq2akZ2dbWvLzMw0PD09jVWrVhmGYRilSpUypkyZYtt/+fJlo0yZMrZrGYZhNGnSxBg6dKhhGIYRHx9vSDKio6NvGOe6desMScaZM2dsbRcvXjSKFi1qbNq0ya5vnz59jC5duhiGYRhjxowxQkJC7PaPHj06x7muV758eWP69Ok33X+9iIgIo1OnTrbP4eHhRkBAgHH+/Hlb2+zZsw1vb28jKysrV7Fff881a9Y0JkyYkOuYAMDVUJEE8qEVK1bI29tbly9fVnZ2trp27aoJEybY9tesWdNuXuSuXbt08OBBFStWzO48Fy9e1KFDh3T27FkdP35cDRo0sO0rXLiw6tevn2N4+5q4uDgVKlTohpW4mzl48KAuXLigli1b2rVfunRJderUkST98ssvdnFIUmhoaK6vcTNvv/22PvzwQyUmJiojI0OXLl1S7dq17fo88MADKlq0qN1109PTdfToUaWnp98y9usNGTJEAwcO1OrVq9WiRQt16tRJtWrV+sf3AgB3CokkkA81a9ZMs2fPlru7u0qXLq3Che1/1b28vOw+p6enq169elq0aFGOc5UsWdJUDNeGqvMiPT1dkrRy5Urdc889dvusVqupOHLj008/1YgRIzR16lSFhoaqWLFieuONN7Rly5Zcn8NM7H379lVYWJhWrlyp1atXKyoqSlOnTtXgwYPN3wwA3EEkkkA+5OXlpSpVquS6f926dfXZZ58pMDBQPj4+N+xTqlQpbdmyRY0bN5YkXblyRdu3b1fdunVv2L9mzZrKzs7W+vXr1aJFixz7r1VEs7KybG0hISGyWq1KTEy8aSWzevXqtgeHrtm8efOtb/Jv/PTTT2rYsKH+85//2NoOHTqUo9+uXbuUkZFhS5I3b94sb29vlS1bVgEBAbeM/UbKli2rAQMGaMCAARozZozmzp1LIgngrsFT2wDUrVs3lShRQh06dNCPP/6ohIQExcTEaMiQIfrjjz8kSUOHDtXrr7+u5cuX69dff9V//vOfv10DskKFCgoPD1fv3r21fPly2zmXLFkiSSpfvrwsFotWrFihEydOKD09XcWKFdOIESM0fPhwLViwQIcOHdKOHTv01ltvacGCBZKkAQMG6MCBAxo5cqTi4+O1ePFizZ8/P1f3+eeffyouLs5uO3PmjKpWraqff/5Zq1at0m+//aZx48Zp27ZtOY6/dOmS+vTpo/379+vbb7/V+PHjNWjQILm5ueUq9usNGzZMq1atUkJCgnbs2KF169apevXquboXAHAJzp6kCeD2+uvDNnnZf/z4caNHjx5GiRIlDKvValSqVMno16+fcfbsWcMwrj5cM3ToUMPHx8fw8/MzIiMjjR49etz0YRvDMIyMjAxj+PDhRqlSpQx3d3ejSpUqxocffmjbP2nSJCM4ONiwWCxGeHi4YRhXHxCaMWOGUa1aNaNIkSJGyZIljbCwMGP9+vW247755hujSpUqhtVqNR599FHjww8/zNXDNpJybAsXLjQuXrxo9OzZ0/D19TX8/PyMgQMHGi+88ILxwAMP5PjeXn75ZaN48eKGt7e30a9fP+PixYu2PreK/fqHbQYNGmRUrlzZsFqtRsmSJY3nnnvOOHny5E3vAQBcjcUwbjJTHgAAAPgbDG0DAADAFBJJAAAAmEIiCQAAAFNIJAEAAGAKiSQAAABMIZEEAACAKSSSAAAAMIVEEgAAAKaQSAIAAMAUEkkAAACYQiIJAAAAU/4f3iKLrLVHTFYAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_confusion_matrix(y_test, y_pred, ['positive', 'negative'], 'NB')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2580FJCGs_oQ",
        "outputId": "118f79e2-6b57-4cc0-a631-c2ef8a7e317e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Classification Report NB:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "    negative       0.86      0.87      0.86      5017\n",
            "    positive       0.87      0.86      0.86      4983\n",
            "\n",
            "    accuracy                           0.86     10000\n",
            "   macro avg       0.86      0.86      0.86     10000\n",
            "weighted avg       0.86      0.86      0.86     10000\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Imprimir as métricas de avaliação\n",
        "print_evaluation_metrics(y_test, y_pred, 'NB')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0JBy6nXvdjC"
      },
      "source": [
        "# Conclusão\n",
        "\n",
        "É possível verificar no relatório de classificação que precisão e recall estão variando entre 86 a 87%. A métrica **F1-Score** combina precisão e recall, possui valor de aproximadamente 86%, o que indica um bom equilíbrio entre precisão e recall. A **Acurácia (accuracy)** geral do modelo é de 86%, o que significa que ele classificou corretamente aproximadamente 86% de todos os exemplos no conjunto de teste.\n",
        "\n",
        "O modelo Naive Bayes com vetorização TF-IDF conseguiu alcançar uma precisão, recall e F1-Score bastante equilibrados para ambas as classes, com uma acurácia geral de 86%. Podemos afirmar que o modelo é capaz de fazer previsões precisas em relação ao sentimento das revisões. Assim, podemos afirmar que o modelo estatístico possui um desempenho consideravelmente superior em relação à abordagem simbólica.\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.11.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}