README.md

---
base_model:
- microsoft/codebert-base
datasets:
- devngho/the_stack_llm_annotations
language:
- code
library_name: transformers
license: mit
metrics:
- f1
---

# devngho/code_edu_classifier_v2_microsoft_codebert-base

이 모델은 [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)에 classifier를 추가한 모델입니다. [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier)의 코드 버전을 목표로, 코드의 교육성 점수를 평가합니다.
학습에는 [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup)에서 추출한 샘플을 [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct)로 평가한 [devngho/the_stack_llm_annotations](https://huggingface.co/datasets/devngho/the_stack_llm_annotations) 데이터셋이 사용되었습니다.

이 연구는 Google의 TPU Research Cloud [(TRC)](https://sites.research.google/trc/about/)의 Cloud TPU 제공으로 수행되었습니다. ⚡

## 상세

- **제작:** devngho
- **언어:** code
- **라이선스:** mit
- **기반 모델:** [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)

## 학습 상세

- learning_rate: 3e-4 (cosine)
- warmup_ratio: 0.1
- batch_size: 2048(512*4)
- optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
- duration: 1h 36m

## 학습 장비

TPU v4-8

## 성능

```
Validation Report:
              precision    recall  f1-score   support

           0       0.77      0.10      0.18       101
           1       0.57      0.47      0.51       739
           2       0.60      0.60      0.60      2409
           3       0.49      0.74      0.59      2030
           4       0.51      0.03      0.05       864
           5       0.00      0.00      0.00         1

    accuracy                           0.54      6144
   macro avg       0.49      0.32      0.32      6144
weighted avg       0.55      0.54      0.50      6144

Confusion Matrix:
[[  10   71   20    0    0    0]
 [   3  346  353   37    0    0]
 [   0  186 1450  770    3    0]
 [   0    9  509 1494   18    0]
 [   0    0   80  762   22    0]
 [   0    0    0    1    0    0]]
```

임베딩 모델이 일부 언어를 지원하지 않는 한계와 qwen2.5 32b 모델의 평가 한계로 성능이 낮은 것으로 보입니다. 3 이상과 미만으로 구분할 때 f1 score는 약 0.77입니다.

# devngho/code_edu_classifier_v2_microsoft_codebert-base

This model is [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) with classfier head. It is designed to evaluate the educational value of codes, similar to the [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier), but focused on code. The training data comes from [devngho/the_stack_llm_annotations](https://huggingface.co/datasets/devngho/the_stack_llm_annotations) dataset, contains samples extracted from [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) and evaluated using [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct).

This research was supported with Cloud TPUs from Google's TPU Research Cloud [(TRC)](https://sites.research.google/trc/about/).⚡

- **Developed by:** devngho
- **Language(s):** code
- **License:** mit
- **Base model:** [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)

## Training detail

- learning_rate: 3e-4 (cosine)
- warmup_ratio: 0.1
- batch_size: 2048(512*4)
- optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
- duration: 3h 21m

## Training hardware

TPU v4-8

## Performance

```
Validation Report:
              precision    recall  f1-score   support

           0       0.77      0.10      0.18       101
           1       0.57      0.47      0.51       739
           2       0.60      0.60      0.60      2409
           3       0.49      0.74      0.59      2030
           4       0.51      0.03      0.05       864
           5       0.00      0.00      0.00         1

    accuracy                           0.54      6144
   macro avg       0.49      0.32      0.32      6144
weighted avg       0.55      0.54      0.50      6144

Confusion Matrix:
[[  10   71   20    0    0    0]
 [   3  346  353   37    0    0]
 [   0  186 1450  770    3    0]
 [   0    9  509 1494   18    0]
 [   0    0   80  762   22    0]
 [   0    0    0    1    0    0]]
```

The low performance is likely due to the limitations of the embedding model, which does not support all languages and the evaluation limitations of the Qwen2.5 32B model. The F1 score is about 0.72 when separating above and below 3.