Update README.md

README.md

@@ -19,7 +19,7 @@ thumbnail: https://github.com/Marcosdib/S2Query/Classification_Architecture_mode
 ![MCTIimg](https://antigo.mctic.gov.br/mctic/export/sites/institucional/institucional/entidadesVinculadas/conselhos/pag-old/RODAPE_MCTI.png)
 
 
-# MCTI Text Classification Task (uncased) DRAFT
+# MCTI Text Classification Task (uncased)
 
 Disclaimer: The Brazilian Ministry of Science, Technology, and Innovation (MCTI) has partially supported this project.
 
@@ -38,24 +38,28 @@ Transformer-based approach, the Word2Vec-based approach improved the accuracy ra
 
 ## Model description
 
-Nullam congue hendrerit turpis et facilisis. Cras accumsan ante mi, eu hendrerit nulla finibus at. Donec imperdiet, 
-nisi nec pulvinar suscipit, dolor nulla sagittis massa, et vehicula ante felis quis nibh. Lorem ipsum dolor sit amet,
-consectetur adipiscing elit. Maecenas viverra tempus risus non ornare. Donec in vehicula est. Pellentesque vulputate 
-bibendum cursus. Nunc volutpat vitae neque ut bibendum:
+After the embedding, which is just essentially data preprocessing, it is necessary to develop the Project 
+further to analyze the input text and classify whether it is a valid research funding opportunity for 
+Brazilian or not. 
 
-- Nullam congue hendrerit turpis et facilisis. Cras accumsan ante mi, eu hendrerit nulla finibus at. Donec imperdiet, 
- nisi nec pulvinar suscipit, dolor nulla sagittis massa, et vehicula ante felis quis nibh. Lorem ipsum dolor sit amet,
- consectetur adipiscing elit.
-- Nullam congue hendrerit turpis et facilisis. Cras accumsan ante mi, eu hendrerit nulla finibus at. Donec imperdiet, 
- nisi nec pulvinar suscipit, dolor nulla sagittis massa, et vehicula ante felis quis nibh. Lorem ipsum dolor sit amet,
- consectetur adipiscing elit.
+For the project, the best option would be chosen empirically upon comparing the results of 4 distinct architectures:
+Neural Network (NN), Deep Neural Network (DNN), Long Short-Term Memory (LSTM), and Convolutional Neural Network (CNN).
+Figure 4 shows the structure of the models.
 
-Nullam congue hendrerit turpis et facilisis. Cras accumsan ante mi, eu hendrerit nulla finibus at. Donec imperdiet, 
-nisi nec pulvinar suscipit, dolor nulla sagittis massa, et vehicula ante felis quis nibh. Lorem ipsum dolor sit amet,
-consectetur adipiscing elit. Maecenas viverra tempus risus non ornare. Donec in vehicula est. Pellentesque vulputate 
-bibendum cursus. Nunc volutpat vitae neque ut bibendum.
+A neural network (NN) here is a simple feedforward neural network with only a single hidden layer, usually called 
+”shallow”. Shallow NNs are often limited in the complexity of the problems they can be trained to solve well. 
 
-![architeru](https://github.com/marcosdib/S2Query/Classification_Architecture_model.png)
+Our CNN model uses a dropout layer feeding into a couple of Conv1D layers and then a MaxPooling layer. After that, 
+we Figure 4: Classification models use a hidden layer composed of a dense layer of size 128, followed by another 
+dropout layer, and finally, the Flatten and final dense classification layer. 
+
+The architecture of the CNN network used is composed of a 50% dropout layer followed by two 1D convolution 
+layers associated with a MaxPooling layer. After max pooling a dense layer of size 128 was added connected 
+to a 50% dropout which finally connects to a flatten layer and the final classification dense layer. Dropout 
+layers help to avoid overfitting the network by masking part of the data so that the network learns to create 
+redundancies in the analysis of the inputs.
+
+![CNN Classification Model](https://raw.githubusercontent.com/chap0lin/WEBIST2022/master/Assets/cnn_model.png)
 
 ## Model variations
 
@@ -74,30 +78,9 @@ Table 1: Templates using Word2Vec and Longformer
 | Longformer | 10.9GB | 
 | Word2Vec | 56.1MB |
 
-
-
-| Keras Embedding + SNN | 92.47 | 88.46 | 79.66 | 100 | 0.2 | 0.7 | 1.8 |
-| Keras Embedding + DNN | 89.78 | 84.41 | 77.81 | 92.57 | 1 | 1.4 | 7.6 |
-| Keras Embedding + CNN | 93.01 | 89.91 | 85.18 | 95.69 | 0.4 | 1.1 | 3.2 |
-| Keras Embedding + LSTM| 93.01 | 88.94 | 83.32 | 95.54 | 1.4 | 2 | 1.8 |
-| Word2Vec + SNN | 89.25 | 83.82 | 74.15 | 97.10 | 1.4 | 1.2 | 9.6 |
-| Word2Vec + DNN | 90.32 | 86.52 | 85.18 | 88.70 | 2 | 6.8 | 7.8 |
-| Word2Vec + CNN | 92.47 | 88.42 | 80.85 | 98.72 | 1.9 | 3.4 | 4.7 |
-| Word2Vec + LSTM | 89.78 | 84.36 | 75.36 | 95.81 | 2.6 | 14.3 | 1.2 |
-| Longformer + SNN | 61.29 | 0 | 0 | 0 | 128 | 1.5 | 36.8 |
-| Longformer + DNN | 91.93 | 87.62 | 80.37 | 97.62 | 81 | 8.4 | 12.7 |
-| Longformer + CNN | 94.09 | 90.69 | 83.41 | 100 | 57 | 4.5 | 9.6 |
-| Longformer + LSTM | 61.29 | 0 | 0 | 0 | 135 | 8.6 | 2.6 |
-
 ## Intended uses
 
-You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
-be fine-tuned on a downstream task. See the [model hub](https://www.google.com) to look for
-fine-tuned versions of a task that interests you.
 
-Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
-to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
-generation you should look at model like XXX.
 
 ### How to use
 
@@ -125,6 +108,15 @@ This model is uncased: it does not make a difference between english and English
 Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
 predictions:
 
+Performance limiting: Loading the longformer model in memory means needing 11Gb available only for the model, 
+without considering the weight of the deep learning network. For training this means we need a 20+ Gb GPU to 
+perform the training. Here this was resolved using the high RAM environment of google Colab Pro and training 
+using CPU which justifies the longer training time per season.
+
+Replicability limitation: Due to the simplicity of the keras embedding model, we are using one hot encoding, 
+and it has a delicate problem for replication in production. This detail is pending further study to define 
+whether it is possible to use one of these models.
+
 -
 -
 This bias will also affect all fine-tuned versions of this model.
@@ -144,14 +136,6 @@ it was coupled to the classification model to train it with the labeled data in
 obtained with related metrics. With this implementation, was reached new levels of accuracy with 86% for CNN architecture 
 and 88% for the LSTM architecture.
 
-Table 6: Results from Pre-trained WE + ML models
-| ML Model | Accuracy | F1 Score | Precision | Recall |
-|:--------:|:---------:|:---------:|:---------:|:---------:|
-| NN | 0.8269 | 0.8545 | 0.8392 | 0.8712 |
-| DNN | 0.7115 | 0.7794 | 0.7255 | 0.8485 |
-| CNN | 0.8654 | 0.9083 | 0.8486 | 0.9773 |
-| LSTM | 0.8846 | 0.9139 | 0.9056 | 0.9318 |
-
 ### Preprocessing
 
 Pre-processing was used to standardize the texts for the English language, reduce the number of insignificant tokens and
@@ -250,9 +234,58 @@ Table 5: Compatibility results (*base = labeled MCTI dataset entries)
 | BBC News Articles | 56.77% |
 | New unlabeled MCTI | 75.26% |
 
-## Evaluation results
+Table 6: Results from Pre-trained WE + ML models
+| ML Model | Accuracy | F1 Score | Precision | Recall |
+|:--------:|:---------:|:---------:|:---------:|:---------:|
+| NN | 0.8269 | 0.8545 | 0.8392 | 0.8712 |
+| DNN | 0.7115 | 0.7794 | 0.7255 | 0.8485 |
+| CNN | 0.8654 | 0.9083 | 0.8486 | 0.9773 |
+| LSTM | 0.8846 | 0.9139 | 0.9056 | 0.9318 |
 
+## Evaluation results
 
+The table below presents the results of accuracy, f1-score, recall and precision obtained in the training of each network. 
+In addition, the necessary times for training each epoch, the data validation execution time and the weight of the deep 
+learning model associated with each implementation were added.
+
+Table 7: Results of experiments
+| Model | Accuracy | F1-score | Recall | Precision | Training time epoch(s) | Validation time (s) | Weight(MB) |
+|------------------------|----------|----------|--------|-----------|------------------------|---------------------|------------|
+| Keras Embedding + SNN | 92.47 | 88.46 | 79.66 | 100.00 | 0.2 | 0.7 | 1.8 |
+| Keras Embedding + DNN | 89.78 | 84.41 | 77.81 | 92.57 | 1.0 | 1.4 | 7.6 |
+| Keras Embedding + CNN | 93.01 | 89.91 | 85.18 | 95.69 | 0.4 | 1.1 | 3.2 |
+| Keras Embedding + LSTM | 93.01 | 88.94 | 83.32 | 95.54 | 1.4 | 2.0 | 1.8 |
+| Word2Vec + SNN | 89.25 | 83.82 | 74.15 | 97.10 | 1.4 | 1.2 | 9.6 |
+| Word2Vec + DNN | 90.32 | 86.52 | 85.18 | 88.70 | 2.0 | 6.8 | 7.8 |
+| Word2Vec + CNN | 92.47 | 88.42 | 80.85 | 98.72 | 1.9 | 3.4 | 4.7 |
+| Word2Vec + LSTM | 89.78 | 84.36 | 75.36 | 95.81 | 2.6 | 14.3 | 1.2 |
+| Longformer + SNN | 61.29 | 0 | 0 | 0 | 128.0 | 1.5 | 36.8 |
+| Longformer + DNN | 91.93 | 87.62 | 80.37 | 97.62 | 81.0 | 8.4 | 12.7 |
+| Longformer + CNN | 94.09 | 90.69 | 83.41 | 100.00 | 57.0 | 4.5 | 9.6 |
+| Longformer + LSTM | 61.29 | 0 | 0 | 0 | 13.0 | 8.6 | 2.6 |
+
+The results obtained surpassed those achieved in goal 6 and goal 9, with the best accuracy obtained of 94% 
+in the longformer + CNN model. We can also observe that the models that achieved the best results were those 
+that used the CNN network for deep learning.
+
+In addition, it was possible to notice that the model of longformer + SNN and longformer + LSTM were not able 
+to learn. Perhaps the models need some adjustments, but each training attempt took between 5 and 8 hours, which 
+made it impossible to try to adjust when other models were already showing promising results.
+
+Above the results obtained, it is also necessary to highlight two limitations found for the replication and 
+training of networks:
+
+
+These 10Gb of the model exceed the Github limit and did not go to the repository, so to run the system we need 
+to download the pre-trained network in the notebook and run the encoder-decoder with the data to create the model. 
+It is advisable to do this in a GPU environment and save the file on the drive. After that change the environment to 
+CPU to perform the training. Trying to generate the model in CPU will take more than 3 hours of processing.
+
+
+The best model that does not have any limitations is Word2Vec + CNN. However, we need to study the limitations to 
+understand whether it is possible to introduce a new model with better accuracy and indicators. These adjustments 
+will be worked on during goals 13 and 14 where the main objective will be to encapsulate the solution in the most 
+suitable way for use in production.
 
 ## Benchmarks