Update README.md

README.md

@@ -33,35 +33,35 @@ models were applied to improve the understanding of each sentence.
 ## According to the abstract,
 
 "The research results serve as a successful case of artificial intelligence in a federal government application".
-More details about the project, architecture model, training model and classifications process can be found in the article
+More details about the project, architecture model, training model, and classifications process can be found in the article
 ["Using transfer learning to classify long unstructured texts with small amounts of labeled data"](https://www.scitepress.org/Link.aspx?doi=10.5220/0011527700003318).
 
 ## Model description
 
 The work consists of a machine learning model with word embedding and Convolutional Neural Network (CNN).
 For the project, a Convolutional Neural Network (CNN) was chosen, as it presents better accuracy in empirical
-comparison with 3 other different architectures: Neural Network (NN), Deep Neural Network (DNN) and Long-Term 
+comparison with 3 other different architectures: Neural Network (NN), Deep Neural Network (DNN), and Long-Term 
 Memory (LSTM).
 
-As the input data is compose of unstructured and nonuniform texts it is essential normalize the data to study 
-little insights and valuable relationships to work with the best features of them. In this way, learning is 
+As the input data is composed of unstructured and nonuniform texts, it is essential to normalize the data to study 
+little insights and valuable relationships to work with their best features. In this way, learning is 
 facilitated and allows the gradient descent to converge more quickly.
 
 The first layer of the model is a pre-trained Word2Vec embedding layer as a method of extracting features from the data that can 
 replace one-hot coding with dimensional reduction. The pre-training of this model is explained further in this document.
 
 After the embedding layer, there is the CNN classification model. The architecture of the CNN network is composed of a 50% dropout layer followed by two 1D convolution layers 
-associated with a MaxPooling layer. After maximum grouping, a dense layer of size 128 is added connected to 
+associated with a MaxPooling layer. After maximum grouping, a dense layer of size 128 is added and connected to 
 a 50% dropout which finally connects to a flattened layer and the final sort dense layer. The dropout layers 
-helped to avoid network overfitting by masking part of the data so that the network learned to create 
-redundancies in the analysis of the inputs.
+helped avoid network overfitting by masking part of the data so that the network learned to create 
+redundancies in analyzing the inputs.
 
 ![CNN Classification Model](https://raw.githubusercontent.com/chap0lin/WEBIST2022/master/Assets/cnn_model.png)
 
 ## Model variations
 
 Table 1 below presents the results of several implementations with different architectures, highlighting the 
-accuracy, f1-score, recall and precision results obtained in the training of each network. 
+accuracy, f1-score, recall, and precision results obtained in each network training. 
 
 Table 1: Results of experiments
 | Model | Accuracy | F1-score | Recall | Precision |
@@ -79,10 +79,10 @@ Table 1: Results of experiments
 | Longformer + CNN | 94.09 | 90.69 | 83.41 | 100.00 |
 | Longformer + LSTM | 61.29 | 0 | 0 | 0 |
 
-Table 2 bellow shows the times required for training each epoch, the data validation execution time and the weight of the deep learning 
+Table 2 below shows the times required for training each epoch, the data validation execution time and the weight of the deep learning 
 model associated with each implementation.
 
-Table 2: Results of Training time epoch, Validation time and Weight
+Table 2: Results of Training time epoch, Validation time, and Weight
 | Model | Training time epoch(s) | Validation time (s) | Weight(MB) |
 |------------------------|:-----------------------:|:-------------------:|:----------:|
 | Keras Embedding + SNN | 100.00 | 0.2 | 0.7 | 1.8 |
@@ -99,15 +99,15 @@ Table 2: Results of Training time epoch, Validation time and Weight
 | Longformer + LSTM | 0 | 13.0 | 8.6 | 2.6 |
 
 In addition, it is possible to notice that the model of Longformer + SNN and Longformer + LSTM were not able 
-to learn. Perhaps the models need some adjustments, however, each training attempt takes between 5 and 8 hours, 
-which made the attempt to adjust unfeasible in view of other models already showing promising results.
+to learn. Perhaps the models need some adjustments; however, each training attempt takes between 5 and 8 hours, 
+which made an attempt to adjust unfeasible because of other models already showing promising results.
 
-With Longformer the problems caused by the size of the model became more visible. First, it was necessary to 
+With Longformer, the problems caused by the size the model's size became more visible. First, it was necessary to 
 actively deallocate unused chunks of memory right after use so that the next steps could be loaded. Then, it 
-was necessary to use a CPU environment for training the networks because the weight of the model exceeded the 
+was necessary to use a CPU environment for training the networks because the model's weight exceeded the 
 16GB of video memory available on the P100 board, available in Colab during training. In this case, the high 
-RAM environment was used, which delivers 25GB of memory for use with the CPU, and this means a longer time 
-required for training since the GPU performs matrix operations faster then a CPU. These models were trained 
+RAM environment was used, which delivers 25GB of memory for use with the CPU, and this means a longer time is
+required for training since the GPU performs matrix operations faster than the CPU. These models were trained 
 5x with 100 training epochs each.
 
 ## Intended uses
@@ -116,10 +116,10 @@ required for training since the GPU performs matrix operations faster then a CPU
 
 ### How to use
 
-This model is available in huggingface spaces to be applied to excel files containing scrapped oportunity data.
+This model is available in Hugging Face spaces to be applied to excel files containing scrapped opportunity data.
 - [NLP MCTI Classification Multi](https://huggingface.co/spaces/unb-lamfo-nlp-mcti/NLP-W2V-CNN-Multi)
 
-You can also find the training and evaluation notebooks in the github repository:
+The training and evaluation notebooks can be found in the github repository:
 - [PPF-MCTI Repository](https://github.com/chap0lin/PPF-MCTI)
 
 
@@ -133,12 +133,12 @@ have little to no wrong encodings and abstract markdowns so that the preprocessi
 Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
 predictions:
 
-Performance limiting: Loading the longformer model in memory means needing 11Gb available only for the model, 
-without considering the weight of the deep learning network. For training this means we need a 20+ Gb GPU to 
+Performance limiting: Loading the longformer model in the memory means needing 11Gb available only for the model 
+without considering the weight of the deep learning network. For training, this means we need a 20+ Gb GPU to 
 perform the training. Here this was resolved using the high RAM environment of google Colab Pro and training 
-using CPU which justifies the longer training time per season.
+using CPU, which justifies the longer training time per season.
 
-Replicability limitation: Due to the simplicity of the keras embedding model, we are using one hot encoding, 
+Replicability limitation: Due to the simplicity of the Keras embedding model, we are using one-hot encoding, 
 and it has a delicate problem for replication in production. This detail is pending further study to define 
 whether it is possible to use one of these models.
 
@@ -146,7 +146,7 @@ This bias will also affect all fine-tuned versions of this model.
 
 ## Training data
 
-The [inputted training data](https://github.com/chap0lin/PPF-MCTI/tree/master/Datasets) was obtained from scrapping techniques, over 30 different platforms e.g. The Royal Society, 
+The [inputted training data](https://github.com/chap0lin/PPF-MCTI/tree/master/Datasets) was obtained from scrapping techniques over 30 different platforms, e.g., The Royal Society, 
 Annenberg foundation, and contained 928 labeled entries (928 rows x 21 columns). Of the data gathered, was used only 
 the main text content (column u). Text content averages 800 tokens in length, but with high variance, up to 5,000 tokens.
 
@@ -156,7 +156,7 @@ the main text content (column u). Text content averages 800 tokens in length, bu
 
 After the pre-trained model of word2vec embeddings had already learned meanings relevant to the classification problem, 
 it was coupled to the classification model to train it with the labeled data in a supervised way. Table 6 shows the results 
-obtained with related metrics. With this implementation, was reached new levels of accuracy with 86% for CNN architecture 
+obtained with related metrics. This implementation reached new levels of accuracy, with 86% for CNN architecture 
 and 88% for the LSTM architecture.
 
 ### Preprocessing
@@ -180,7 +180,7 @@ Table 3: Python packages used
 |--------------------------------------------------------|--------------|
 | Resolve contractions and slang usage in text | [contractions](https://pypi.org/project/contractions) |
 | Natural Language Processing | [nltk](https://pypi.org/project/nltk) |
-| Others data manipulations and calculations included in Python 3.10: io, json, math, re (regular expressions), shutil, time, unicodedata; | [numpy](https://pypi.org/project/numpy) |
+| Other data manipulations and calculations included in Python 3.10: io, json, math, re (regular expressions), shutil, time, unicodedata; | [numpy](https://pypi.org/project/numpy) |
 | Data manipulation and analysis | [pandas](https://pypi.org/project/pandas) |
 | http library | [requests](https://pypi.org/project/requests) |
 | Training model | [scikit-learn](https://pypi.org/project/scikit-learn) |
@@ -211,7 +211,7 @@ evaluation of the first four bases (xp1, xp2, xp3, xp4).
 Then, the content simplification was evaluated, from the xp4 base, considering stemming (xp5), Lemmatization (xp6), 
 stemming + stopwords removal (xp7), and Lemmatization + stopwords removal (xp8).
 
-All eight bases were evaluated to classify the eligibility of the opportunity, through the training of a shallow 
+All eight bases were evaluated to classify the eligibility of the opportunity through the training of a shallow 
 neural network (SNN – Shallow Neural Network). The metrics for the eight bases were evaluated. The results are 
 shown in Table 5.
 
@@ -229,20 +229,20 @@ Table 5: Results obtained in Preprocessing
 | xp8 | ap4 + Lemmatization + Stopwords Removal | 92,47% | 88,46% | 79,66% | 100,00% | 225,580 | 16081 | 2726 |
 
 Even so, between these two excellent options, one can judge which one to choose. XP7: It has less training time, 
-less number of unique tokens. XP8: It has smaller maximum sizes. In this case, the criterion used for the choice 
+and less number of unique tokens. XP8: It has smaller maximum sizes. In this case, the criterion used for the choice 
 was the computational cost required to train the vector representation models (word-embedding, sentence-embeddings, 
 document-embedding). The training time is so close that it did not have such a large weight for the analysis.
 
-As a last step, a spreadsheet was generated for the model (xp8) with the fields opo_pre and opo_pre_tkn, containing the
+As the last step, a spreadsheet was generated for the model (xp8) with the fields opo_pre and opo_pre_tkn, containing the
 preprocessed text in sentence format and tokens, respectively. This [database](https://github.com/mcti-sefip/mcti-sefip-ppfcd2020/blob/pre-processamento/Pre_Processamento/oportunidades_final_pre_processado.xlsx) was made 
 available on the project's GitHub with the inclusion of columns opo_pre (text) and opo_pre_tkn (tokenized). 
 
 ### Pretraining
 
-Since labeled data is scarce, word-embeddings was trained in an unsupervised manner using other datasets that 
+Since labeled data is scarce, word-embeddings were trained in an unsupervised manner using other datasets that 
 contain most of the words it needs to learn. The idea implemented was based on introducing better and better-trained 
 word embeddings in the model. For an additional dataset to be applied to improve word-embedding training, it must be
-compatible with the dataset used to train the classifier. Was searched for datasets from the Kaggle, a platform with
+compatible with the dataset used to train the classifier. We searched for datasets from the Kaggle, a platform with
 over a thousand available NLP datasets, and the closest we found was the BBC News Articles dataset, which achieved 
 only 56% compatibility.
 
@@ -267,8 +267,8 @@ Table 7: Results from Pre-trained WE + ML models
 
 ## Evaluation results
 
-The table below presents the results of accuracy, f1-score, recall and precision obtained in the training of each network. 
-In addition, the necessary times for training each epoch, the data validation execution time and the weight of the deep 
+The table below presents the results of accuracy, f1-score, recall, and precision obtained in the training of each network. 
+In addition, the necessary times for training each epoch, the data validation execution time, and the weight of the deep 
 learning model associated with each implementation were added.
 
 Table 8: Results of experiments
@@ -293,7 +293,7 @@ that used the CNN network for deep learning.
 
 With the motivation to increase accuracy obtained with baseline implementation, was implemented a transfer learning 
 strategy under the assumption that small data available for training was insufficient for adequate embedding training. 
-In this context, was considered two approaches: 
+In this context, were considered two approaches: 
 
  - Pre-training word embeddings using similar datasets for text classification;
  - Using transformers and attention mechanisms (Longformer) to create contextualized embeddings. 
@@ -314,15 +314,15 @@ Above the results obtained, it is also necessary to highlight two limitations fo
 training of networks:
 
 
-These 10Gb of the model exceed the Github limit and did not go to the repository, so to run the system we need 
+These 10Gb of the model exceeded the Github limit and did not go to the repository, so to run the system, we need 
 to download the pre-trained network in the notebook and run the encoder-decoder with the data to create the model. 
-It is advisable to do this in a GPU environment and save the file on the drive. After that change the environment to 
-CPU to perform the training. Trying to generate the model in CPU will take more than 3 hours of processing.
+It is advisable to do this in a GPU environment and save the file on the drive. After that, change the environment to 
+CPU to perform the training. Trying to generate the model unsing the CPU will take more than 3 hours of processing.
 
 
 The best model that does not have any limitations is Word2Vec + CNN. However, we need to study the limitations to 
 understand whether it is possible to introduce a new model with better accuracy and indicators. These adjustments 
-will be worked on during goals 13 and 14 where the main objective will be to encapsulate the solution in the most 
+will be worked on during goals 13 and 14, where the main objective will be to encapsulate the solution in the most 
 suitable way for use in production.
 
 ## Benchmarks