File size: 3,052 Bytes
9a42a11 c914a4d c1d8ae6 c914a4d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
---
license: mit
---
# FinGPT sentiment analysis task
## Model info
- Base model: InternLM-20B
- Training method: Instruction Fine-tuning + LoRA
- Task: Sentiment Analysis
## Packages
``` python
!pip install transformers==4.32.0 peft==0.5.0
!pip install sentencepiece
!pip install accelerate
!pip install torch
!pip install peft
!pip install datasets
!pip install bitsandbytes
```
## Inference: Try the model in Google Colab
``` python
from transformers import AutoModel, AutoTokenizer, AutoModelForCausalLM, LlamaForCausalLM, LlamaTokenizerFast
from peft import PeftModel # 0.5.0
# Load Models
base_model = "internlm/internlm-20b"
peft_model = "FinGPT/fingpt-sentiment_internlm-20b_lora"
tokenizer = LlamaTokenizerFast.from_pretrained(base_model, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
model = LlamaForCausalLM.from_pretrained(base_model, trust_remote_code=True, device_map = "cuda:0", load_in_8bit = True,)
model = PeftModel.from_pretrained(model, peft_model)
model = model.eval()
# Make prompts
prompt = [
'''Instruction: What is the sentiment of this news? Please choose an answer from {negative/neutral/positive}
Input: FINANCING OF ASPOCOMP 'S GROWTH Aspocomp is aggressively pursuing its growth strategy by increasingly focusing on technologically more demanding HDI printed circuit boards PCBs .
Answer: ''',
'''Instruction: What is the sentiment of this news? Please choose an answer from {negative/neutral/positive}
Input: According to Gran , the company has no plans to move all production to Russia , although that is where the company is growing .
Answer: ''',
'''Instruction: What is the sentiment of this news? Please choose an answer from {negative/neutral/positive}
Input: A tinyurl link takes users to a scamming site promising that users can earn thousands of dollars by becoming a Google ( NASDAQ : GOOG ) Cash advertiser .
Answer: ''',
]
# Generate results
tokens = tokenizer(prompt, return_tensors='pt', padding=True, max_length=512)
res = model.generate(**tokens, max_length=512)
res_sentences = [tokenizer.decode(i) for i in res]
out_text = [o.split("Answer: ")[1] for o in res_sentences]
# show results
for sentiment in out_text:
print(sentiment)
# Output:
# positive
# neutral
# negative
```
## Training Script: [Our Code](https://github.com/AI4Finance-Foundation/FinGPT/tree/master/fingpt/FinGPT_Benchmark)
```
#internlm-20b
deepspeed -i "localhost:2" train_lora.py
--run_name sentiment-internlm-20b-8epochs-lr2e-4-linear
--base_model internlm-20b
--dataset data/fingpt-sentiment-train
--max_length 512
--batch_size 8
--learning_rate 2e-4
--num_epochs 8
> train_internlm-20b_1gpu_8epochs_lr2e4_bs8_fp16_linear.log 2>&1
```
## inference script
```
CUDA_VISIBLE_DEVICES=1 python benchmarks.py \
--dataset fpb,fiqa,tfns,nwgi \
--base_model internlm-20b \
--peft_model FinGPT/fingpt-sentiment_internlm-20b_lora \
--batch_size 1 \
--max_length 512 \
--from_remote True
```
## Training Data:
* https://huggingface.co/datasets/FinGPT/fingpt-sentiment-train
- PEFT 0.5.0
|