AdaptLLM commited on
Commit
778eb88
·
verified ·
1 Parent(s): d1aafbb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +6 -16
README.md CHANGED
@@ -18,40 +18,30 @@ We investigate domain adaptation of MLLMs through post-training, focusing on dat
18
  <img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/Xsz4p6f6BVi7XqIk8eUNu.png" width="900">
19
  </p>
20
 
21
- ### **Updates**
22
  - **[2024/11/29]** Released our paper.
23
 
24
 
25
  ## About
26
 
27
- AdaMLLM is our latest effort to enhance task generalization of (M)LLMs by scaling synthetic supervised tasks based on unsupervised contexts.
28
 
29
  <p align='left'>
30
- <img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/HUN3Cr66w_xpj5_c7QQaI.png" width="1000">
31
  </p>
32
 
33
  - [AdaptLLM](https://huggingface.co/papers/2309.09530)
34
  We employ rule-based methods to extract tasks from domain-specific corpora, reformatting them into reading comprehension tasks for continued pre-training. Our 7B finance model outperforms domain-specific models of much larger scales, such as BloombergGPT-50B.
35
 
36
- - [Instruction Pre-Training](https://huggingface.co/papers/2406.14491)
37
- We develop a general-purpose instruction synthesizer which significantly increases task diversity for LM pre-training, outperforming vanilla pre-training in both general pre-training from scratch and domain-adaptive continual pre-training.
38
-
39
  - AdaMLLM
40
- We extend supervised task synthesis to multimodality, introducing a unified visual instruction synthesizer to extract instruction-response pairs from image-caption data. Our synthetic tasks outperform those generated by manual rules, GPT-4, and GPT-4V in improving domain-specific performance for MLLMs.
41
-
42
- Looking ahead, we envision further broadening the scope of supervised task synthesis, efficiently enhancing the general capabilities of trained models.
43
 
44
 
45
  ## Citation
46
  If you find our work helpful, please consider citing us.
47
- ```bibtex
48
- @article{instructPT,
49
- title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
50
- author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu},
51
- journal={arXiv preprint arXiv:2406.14491},
52
- year={2024}
53
- }
54
 
 
 
55
  @inproceedings{
56
  adaptllm,
57
  title={Adapting Large Language Models via Reading Comprehension},
 
18
  <img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/Xsz4p6f6BVi7XqIk8eUNu.png" width="900">
19
  </p>
20
 
21
+ ### Updates
22
  - **[2024/11/29]** Released our paper.
23
 
24
 
25
  ## About
26
 
27
+ AdaMLLM represents our latest advancement in building domain-specific foundation models through post-training on synthetic supervised tasks derived from unsupervised contexts.
28
 
29
  <p align='left'>
30
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/650801ced5578ef7e20b33d4/R3WoJXdGsjLTfoqJ93fVI.png" width="800">
31
  </p>
32
 
33
  - [AdaptLLM](https://huggingface.co/papers/2309.09530)
34
  We employ rule-based methods to extract tasks from domain-specific corpora, reformatting them into reading comprehension tasks for continued pre-training. Our 7B finance model outperforms domain-specific models of much larger scales, such as BloombergGPT-50B.
35
 
 
 
 
36
  - AdaMLLM
37
+ We extend supervised task synthesis to multimodality, introducing a unified visual instruction synthesizer to extract instruction-response pairs from domain-specific image-caption pairs. Our synthetic tasks outperform those generated by manual rules, GPT-4, and GPT-4V in improving domain-specific performance for MLLMs.
 
 
38
 
39
 
40
  ## Citation
41
  If you find our work helpful, please consider citing us.
 
 
 
 
 
 
 
42
 
43
+ [AdaptLLM](https://huggingface.co/papers/2309.09530)
44
+ ```bibtex
45
  @inproceedings{
46
  adaptllm,
47
  title={Adapting Large Language Models via Reading Comprehension},