PEFT
English
music
ThatOneShortGuy commited on
Commit
6bbc271
1 Parent(s): e5b9bb5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -5
README.md CHANGED
@@ -1,20 +1,91 @@
1
  ---
2
  library_name: peft
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  ## Training procedure
5
 
6
 
7
- The following `bitsandbytes` quantization config was used during training:
8
- - load_in_8bit: True
9
  - load_in_4bit: False
10
  - llm_int8_threshold: 6.0
11
  - llm_int8_skip_modules: None
12
- - llm_int8_enable_fp32_cpu_offload: False
13
  - llm_int8_has_fp16_weight: False
14
  - bnb_4bit_quant_type: fp4
15
  - bnb_4bit_use_double_quant: False
16
- - bnb_4bit_compute_dtype: float32
17
  ### Framework versions
18
 
19
 
20
- - PEFT 0.5.0.dev0
 
1
  ---
2
  library_name: peft
3
+ license: apache-2.0
4
+ datasets:
5
+ - ThatOneShortGuy/SongLyrics
6
+ language:
7
+ - en
8
+ tags:
9
+ - music
10
  ---
11
+ # Musical Falcon
12
+
13
+ [OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000) model fine tuned using PEFT on
14
+ [Song Lyrics](https://huggingface.co/datasets/ThatOneShortGuy/SongLyrics) to write lyrics to songs.
15
+
16
+ ## Model Details
17
+ - **Finetuned from**: [OpenAssistant/falcon-7b-sft-mix-2000]
18
+ - **Model Type**: Causal decoder-only transformer language model
19
+ - **Language**: English (and limited capabilities in German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);
20
+ - **License**: Apache 2.0
21
+ - **Contact**: Lol don't. This is just for fun.
22
+
23
+ ## Usage
24
+ The basic basic format getting it in is:
25
+ ```python
26
+ from peft import PeftModel, PeftConfig
27
+ from transformers import AutoModelForCausalLM
28
+
29
+ config = PeftConfig.from_pretrained("ThatOneShortGuy/MusicalFalcon")
30
+ model = AutoModelForCausalLM.from_pretrained("OpenAssistant/falcon-7b-sft-mix-2000")
31
+ model = PeftModel.from_pretrained(model, "ThatOneShortGuy/MusicalFalcon")
32
+ ```
33
+
34
+ ## Prompting
35
+ Considering this comes from [OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000), it uses the same structure.
36
+ Two special tokens are used to mark the beginning of user and assistant turns: `<|prompter|>` and `<|assistant|>`. Each turn ends with a `<|endoftext|>` token.
37
+ The training prompt used the structure:
38
+ ```
39
+ <|prompter|>Come up with the lyrics for a song from "{artist}" {"from " + year if year else ""} titled "{title}".<|endoftext|>
40
+ <|assistant|>Sure! Here are the lyrics:
41
+ {lyrics}
42
+ <|endoftext|>
43
+ ```
44
+
45
+ However, it still seems to work just fine using:
46
+ ```
47
+ <|prompter|>Write me a song titled "{title}"<|endoftext|><|assistant|>
48
+ ```
49
+ or any anything of similar nature. Feel free to add a description of the song in there too.
50
+
51
+ ## Examples
52
+ Prompt
53
+ ```
54
+ <|prompter|>Write a song with the lyrics for a song from artist "Eminem" from 2018 titled "Suzy's Lemonade Stand"<|endoftext|>
55
+ <|assistant|>Sure! Here are the lyrics:
56
+ ```
57
+ Result
58
+ ```
59
+ [Intro]
60
+ I'm gonna make you famous, baby
61
+
62
+ [Verse 1]
63
+ It's been a long time since I had a lemonade stand
64
+ But I'm back in town and I'ma do it again
65
+ You know what? I've been on my own since I was born
66
+ So I'll be here all day long, so come on, let's go
67
+ I got some new tricks up my sleeve that I'd like to show
68
+ And I'mma get your name out of your mouth, so please don't blow
69
+ Let me see if I can get you to say my name
70
+ If not, then I'l just have to take it back to the drawing board
71
+ I gotta get this money, I's a little bit broke
72
+ I need some cash, I need some dough, I gotta get these clothes off
73
+ I wanna be a star, I wanna be the man
74
+ I want to be a superstar, I want to get my hands on them bands
75
+ ```
76
+
77
+
78
  ## Training procedure
79
 
80
 
 
 
81
  - load_in_4bit: False
82
  - llm_int8_threshold: 6.0
83
  - llm_int8_skip_modules: None
84
+ - llm_int8_enable_fp32_cpu_offload: True
85
  - llm_int8_has_fp16_weight: False
86
  - bnb_4bit_quant_type: fp4
87
  - bnb_4bit_use_double_quant: False
 
88
  ### Framework versions
89
 
90
 
91
+ - PEFT 0.5.0.dev0