KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
Paper
•
2412.06071
•
Published
•
7
John
juyongjiang
AI & ML interests
LLM for Code Generation
Recent Activity
upvoted
a
paper
14 days ago
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
Organizations
juyongjiang's activity
upvoted
a
paper
14 days ago
upvoted
an
article
4 months ago
Article
Introducing the Open Ko-LLM Leaderboard: Leading the Korean LLM Evaluation Ecosystem
•
3
liked
a
model
4 months ago
reacted to
chansung's
post with 🚀➕🔥👍
4 months ago
Post
4012
🦙🦙 LLaMA Duo project update
Last time, I gave a brief introduction about LLaMA Duo project with @sayakpaul . It is a simple toolset to aligning sLLM with service LLM with coverage dataset 👉🏻 (https://huggingface.co/posts/chansung/708646454991943).
- coverage dataset is what we believe to be the most important/desired (instruction, response) pairs. In system thinking, each instruction could be an analogy of a function from traditional programming. We make unit tests and measure the coverage % for all the features/functions. Similarly, we need to ensure if our fine-tuned model could handle what % of given instructions from coverage dataset satisfactory (hence coverage dataset).
We have tested it with "Coding" category of data from HuggingFaceH4/no_robots dataset. It has about 300 SFT training data points under Coding category. After fine-tuning Gemma 7B model on that, the result was very poor. LLaMA Duo's evaluation tool gave < 20% of metrics in similarity and preciseness on the test split.
So, we used LLaMA Duo's synthetic data generation tool to generate 60k data points that looks similar to the original dataset. We first created ~10k synthetic data points, then created 50k more based on the synthetic dataset itself.
After fine-tuning Gemma 7B on the 60k synthetic dataset, the evaluation result went up to 80~90% high. Also, when testing out the model in UI, it tends to give good responses.
It is a good showcase to transition from service LLM to sLLM or having a backup sLLM for service LLM failure scenarios. I am going to expand this experiments on all categories of no_robots dataset. It will roughly generate > 100k data points.
Here are some links:
- LLaMA Duo project repo: https://github.com/deep-diver/llamaduo
- 60k Coding synthetic dataset: chansung/merged_ds_coding
- Fine-tuned Gemma 7B model: chansung/coding_llamaduo_60k_v0.2
Last time, I gave a brief introduction about LLaMA Duo project with @sayakpaul . It is a simple toolset to aligning sLLM with service LLM with coverage dataset 👉🏻 (https://huggingface.co/posts/chansung/708646454991943).
- coverage dataset is what we believe to be the most important/desired (instruction, response) pairs. In system thinking, each instruction could be an analogy of a function from traditional programming. We make unit tests and measure the coverage % for all the features/functions. Similarly, we need to ensure if our fine-tuned model could handle what % of given instructions from coverage dataset satisfactory (hence coverage dataset).
We have tested it with "Coding" category of data from HuggingFaceH4/no_robots dataset. It has about 300 SFT training data points under Coding category. After fine-tuning Gemma 7B model on that, the result was very poor. LLaMA Duo's evaluation tool gave < 20% of metrics in similarity and preciseness on the test split.
So, we used LLaMA Duo's synthetic data generation tool to generate 60k data points that looks similar to the original dataset. We first created ~10k synthetic data points, then created 50k more based on the synthetic dataset itself.
After fine-tuning Gemma 7B on the 60k synthetic dataset, the evaluation result went up to 80~90% high. Also, when testing out the model in UI, it tends to give good responses.
It is a good showcase to transition from service LLM to sLLM or having a backup sLLM for service LLM failure scenarios. I am going to expand this experiments on all categories of no_robots dataset. It will roughly generate > 100k data points.
Here are some links:
- LLaMA Duo project repo: https://github.com/deep-diver/llamaduo
- 60k Coding synthetic dataset: chansung/merged_ds_coding
- Fine-tuned Gemma 7B model: chansung/coding_llamaduo_60k_v0.2
upvoted
a
paper
4 months ago
authored
a
paper
4 months ago
liked
a
dataset
5 months ago
updated
10
models
7 months ago
llama-duo/gemma7b-summarize-claude3sonnet-256k
Updated
llama-duo/gemma7b-summarize-gemini1_5flash-64k
Updated
•
2
llama-duo/gemma7b-summarize-gemini1_5flash-32k
Updated
•
1
llama-duo/gemma7b-summarize-gpt4o-16k
Updated
llama-duo/gemma7b-summarize-gpt4o-8k
Updated
•
3
llama-duo/gemma7b-summarize-gpt4o-4k
Updated
•
3
llama-duo/gemma7b-summarize-gpt4o-2k
Updated
•
2
llama-duo/gemma7b-summarize-gpt4o-1k
Updated
•
2
llama-duo/gemma7b-summarize-claude3sonnet-128k
Updated
•
2
llama-duo/gemma7b-summarize-claude3sonnet-64k
Updated