kimsan0622
/

Llama-3.2-1B-Code-Knowledge-Value-Eval

@@ -1,199 +1,110 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 library_name: transformers
+license: llama3.2
+base_model: meta-llama/Llama-3.2-1B
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: code-knowledge-eval
+  results: []
 ---
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+# **Llama-3.2-1B-Code-Knowledge-Value-Eval **
+This model is a fine-tuned version of [meta-llama/Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B) on the [kimsan0622/code-knowledge-eval](https://huggingface.co/datasets/kimsan0622/code-knowledge-eval) dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.0447
+- Accuracy: 0.5227
+## **Model Description**
+The model trained on the **Code Knowledge Value Evaluation Dataset** is designed to assess the educational and knowledge value of code snippets. It leverages patterns and contextual information from a large collection of open-source code, sourced from the `bigcode/the-stack` repository. By analyzing these code samples, the model can evaluate their utility in teaching coding concepts, solving problems, and improving developer education.
+The model focuses on understanding the structure, syntax, and logic of various programming languages, enabling it to provide insights into the learning potential and technical depth of different code samples. The dataset used for training consists of 22,786 samples for training, 4,555 for validation, and 18,232 for testing, ensuring that the model is both robust and well-generalized across different coding contexts.
+## **Intended Uses & Limitations**
+### **Intended Uses**:
+1. **Automated Code Review**: The model can be applied in automated systems to assess the knowledge value of code during code review processes. It can help identify areas where code could be optimized for better readability, maintainability, and educational impact.
+2. **Educational Feedback**: For instructors and educational platforms, the model can offer feedback on the effectiveness of code samples used in teaching, helping to improve curriculum materials and select code that best conveys core programming concepts.
+3. **Curriculum Development**: The model can aid in designing coding courses or instructional materials by suggesting code examples that have higher educational value, supporting a more effective learning experience.
+4. **Technical Skill Assessment**: Organizations or platforms can use the model to assess the complexity and educational value of code submissions in coding challenges or exams.
+### **Limitations**:
+1. **Narrow Scope in Knowledge Evaluation**: The model is specialized in evaluating code from an educational standpoint, focusing primarily on learning potential rather than production-level code quality (e.g., performance optimization or security).
+2. **Language and Domain Limitations**: Since the dataset is sourced from `bigcode/the-stack`, it may not cover all programming languages or specialized domains. The model may perform less effectively in underrepresented languages or niche coding styles not well-represented in the dataset.
+3. **Not Suitable for All Educational Levels**: While the model is designed to evaluate code for educational purposes, its outputs may be better suited for certain levels (e.g., beginner or intermediate coding), and its recommendations might not fully cater to advanced or highly specialized learners.
+## Training and evaluation data
+[kimsan0622/code-knowledge-eval](https://huggingface.co/datasets/kimsan0622/code-knowledge-eval)
+## Training procedure
+### Training hyperparameters
+The following hyperparameters were used during training:
+- learning_rate: 0.0001
+- train_batch_size: 128
+- eval_batch_size: 64
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 5.0
+### Training results
+| Training Loss | Epoch | Step | Validation Loss | Accuracy |
+|:-------------:|:-----:|:----:|:---------------:|:--------:|
+| 1.0722        | 1.0   | 179  | 1.2996          | 0.4296   |
+| 0.9632        | 2.0   | 358  | 1.0447          | 0.5227   |
+| 0.6905        | 3.0   | 537  | 1.1845          | 0.5412   |
+| 0.3417        | 4.0   | 716  | 1.7654          | 0.5317   |
+| 0.1224        | 5.0   | 895  | 2.3238          | 0.5322   |
+### Framework versions
+- Transformers 4.44.2
+- Pytorch 2.3.0
+- Datasets 2.20.0
+- Tokenizers 0.19.1
+## Test set results
+### Confusion matrix
+| y_true |**pred_0**|**pred_1**|**pred_2**|**pred_3**|**pred_4**|**pred_5**|
+|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
+|   0    |  595  |  680  |   21  |   70  |   11  |   0   |
+|   1    |  116  |  727  |   76  |  283  |   39  |   3   |
+|   2    |   36  |  542  |  120  |  871  |  187  |  18   |
+|   3    |   14  |  287  |  107  | 2387  | 1958  |  172  |
+|   4    |   3   |   42  |    8  |  924  | 3798  | 1549  |
+|   5    |   0   |    3  |    0  |   20  |  583  | 1982  |
+### Classification report
+|  **y_true**   | **precision** | **recall** | **f1-score** | **support** |
+|:-------------:|:-------------:|:----------:|:------------:|:-----------:|
+|       0       |     0.78      |    0.43    |     0.56     |    1377     |
+|       1       |     0.32      |    0.58    |     0.41     |    1244     |
+|       2       |     0.36      |    0.07    |     0.11     |    1774     |
+|       3       |     0.52      |    0.48    |     0.50     |    4925     |
+|       4       |     0.58      |    0.60    |     0.59     |    6324     |
+|       5       |     0.53      |    0.77    |     0.63     |    2588     |
+|  **accuracy** |               |            |     0.53     |   18232     |
+|  **macro avg**|     0.52      |    0.49    |     0.47     |   18232     |
+| **weighted avg** |  0.53      |    0.53    |     0.51     |   18232     |