twinkle-ai/Llama-3.2-3B-F1-Instruct
Text Generation
•
Updated
•
6
•
14
Huang Liang Hsun PRO
lianghsun
AI & ML interests
Founder of Twinkle AI. Focused on applying deep learning in legal and scientific domains, with expertise in NLP and model fine-tuning.
Recent Activity
updated
a dataset
about 12 hours ago
twinkle-ai/tw-reasoning-instruct-50k
updated
a model
about 16 hours ago
twinkle-ai/Llama-3.2-3B-F1-Instruct
replied to
their
post
1 day ago
With the arrival of Twinkle April — Twinkle AI’s annual open-source celebration held every April — our community is excited to unveil its very first project:
📊 Twinkle Eval (https://github.com/ai-twinkle/Eval), a next-generation evaluation tool led by our contributor @tedslin .
Unlike traditional evaluation tools like iKala’s ievals (https://github.com/ikala-ai/ievals), which can only evaluate language models (LMs) one sample at a time, Twinkle Eval is designed with Large Reasoning Models (LRMs) in mind. As reasoning time increases with more complex models, traditional tools become increasingly inefficient 😲 — for example, evaluating LRMs on the https://huggingface.co/datasets/ikala/tmmluplus benchmark could take *
half a day without finishing.
One question we were especially curious about:
Does shuffling multiple-choice answer order impact model accuracy? 🤔
→ See: "Change Answer Order Can Decrease MMLU Accuracy" – arXiv:2406.19470v1
To address these challenges, Twinkle Eval brings three key innovations to the table:
1️⃣ Parallelized evaluation of samples
2️⃣ Multi-round testing for stability
3️⃣ Randomized answer order to test robustness
After running experiments, we observed that Twinkle Eval can speed up evaluation by up to 15× 🚀🚀. Interestingly, most models scored slightly lower under the 2️⃣3️⃣ test settings compared to their claimed performance — suggesting further benchmarking is needed.
This framework also comes with additional tunable parameters and detailed logging of LM behavior per question — perfect for those who want to dive deeper. 😆
If you find Twinkle Eval useful, please ⭐ the project and help spread the word 🤗
Organizations
lianghsun's activity
liked
a
model
5 days ago
liked
a
dataset
5 days ago
liked
a
model
7 days ago
liked
3
datasets
8 days ago
liked
2
datasets
11 days ago
liked
a
dataset
16 days ago
liked
7
datasets
17 days ago
tyouisen/aclue
Updated
•
137
•
8
CohereForAI/Global-MMLU
Viewer
•
Updated
•
602k
•
18.8k
•
116
tidarren/ptt-riddle
Viewer
•
Updated
•
17.1k
•
31
•
1
STEM-AI-mtl/City_map
Viewer
•
Updated
•
605
•
37
•
4
milkshake721/2.1M-wiki-STEM
Viewer
•
Updated
•
2.1M
•
55
•
3
fmars/wiki_stem
Viewer
•
Updated
•
676k
•
117
•
5
Abzu/arxiv_stem
Viewer
•
Updated
•
2.3M
•
177
•
1
liked
a
dataset
22 days ago
liked
a
dataset
24 days ago
liked
2
datasets
29 days ago