IKT_classifier_target_best
This model is a fine-tuned version of sentence-transformers/all-mpnet-base-v2 on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.2820
- Precision Macro: 0.9487
- Precision Weighted: 0.9455
- Recall Macro: 0.9378
- Recall Weighted: 0.9423
- F1-score: 0.9412
- Accuracy: 0.9423
Model description
The model is a binary text classifier based on sentence-transformers/all-mpnet-base-v2 and fine-tuned on text sourced from national climate policy documents.
Intended uses & limitations
The classifier assigns a class of 'Target' or 'Negative' to denote alignment with stated national targets as portrayed in extracted passages from the documents. The intended use is for climate policy researchers and analysts seeking to automate the process of reviewing lengthy, non-standardized PDF documents to produce summaries and reports.
The performance of the classifier is very high. On training, the classifier exhibited very good overall performance (F1 ~ 0.95). This performance was evenly balanced between precise identification of true positive classifications (precision ~ 0.95) and a wide net to capture as many true positives as possible (recall ~ 0.95). When tested on real world unseen test data, the performance was still very high (F1 ~ 0.9). However, testing was based on a fairly small out-of-sample dataset. Therefore classification performance will need to further evaluated on deployment.
Training and evaluation data
The training dataset is comprised of labelled passages from 2 sources:
- ClimateWatch NDC Sector data.
- IKI TraCS Climate Strategies for Transport Tracker implemented by GIZ and funded by theInternational Climate Initiative (IKI) of the German Federal Ministry for Economic Affairs and Climate Action (BMWK). Here we utilized the QA dataset (CW_NDC_data_Sector).
The combined datasetGIZ/policy_qa_v0_1 contains ~85k rows. Each row is duplicated twice, to provide varying sequence lengths (denoted by the values 'small', 'medium', and 'large', which correspond to sequence lengths of 60, 85, and 150 respectively - indicated in the 'strategy' column). This effectively means the dataset is reduced by 1/3 in useful size, and the 'strategy' value should be selected based on the use case. For this training, we utilized the 'medium' samples Furthermore, for each row, the 'context' column contains 3 samples of varying quality. The approach used to assess quality and select samples is described below.
The pre-processing operations used to produce the final training dataset were as follows:
- Dataset is filtered based on 'medium' value in 'strategy' column (sequence length = 85).
- For ClimateWatch, all rows are removed as there was assessed to be no taxonomical alignment with the IKITracs labels inherent to the dataset.
- For IKITracs, labels are assigned based on the presence of certain substring prefixes ('T_') based on 'parameter' values which correspond to text containing targets as assessed by human annotaters.
- If 'context_translated' is available and the 'language' is not English, 'context' is replaced with 'context_translated'. This results in the model being trained on English translations of original text samples.
- The dataset is "exploded" - i.e., the text samples in the 'context' column, which are lists, are converted into separate rows - and labels are merged to align with the associated samples.
- The 'match_onanswer' and 'answerWordcount' are used conditionally to select high quality samples (prefers high % of word matches in 'match_onanswer', but will take lower if there is a high 'answerWordcount')
- No data augmentation was conducted as the number of samples were high for the 'TARGET' class. The end result is an equal sample per class breakdown of:
- TARGET: 777
- NEGATIVE: 778
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 4.055539898211945e-05
- train_batch_size: 3
- eval_batch_size: 3
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 400.0
- num_epochs: 5
Training results
Training Loss | Epoch | Step | Validation Loss | Precision Macro | Precision Weighted | Recall Macro | Recall Weighted | F1-score | Accuracy |
---|---|---|---|---|---|---|---|---|---|
No log | 1.0 | 467 | 0.3171 | 0.9197 | 0.9179 | 0.9131 | 0.9167 | 0.9154 | 0.9167 |
0.4991 | 2.0 | 934 | 0.2515 | 0.9197 | 0.9179 | 0.9131 | 0.9167 | 0.9154 | 0.9167 |
0.4101 | 3.0 | 1401 | 0.2636 | 0.9407 | 0.9381 | 0.9319 | 0.9359 | 0.9348 | 0.9359 |
0.2713 | 4.0 | 1868 | 0.2515 | 0.9487 | 0.9455 | 0.9378 | 0.9423 | 0.9412 | 0.9423 |
0.2038 | 5.0 | 2335 | 0.2820 | 0.9487 | 0.9455 | 0.9378 | 0.9423 | 0.9412 | 0.9423 |
Framework versions
- Transformers 4.31.0
- Pytorch 2.0.1+cu118
- Datasets 2.13.1
- Tokenizers 0.13.3
- Downloads last month
- 18
Model tree for mtyrrell/CPU_Target_Classifier
Base model
sentence-transformers/all-mpnet-base-v2