| # Model Card for GPT_2_CODE | |
| <!-- Provide a quick summary of what the model is/does. [Optional] --> | |
| WIP, | |
| Goal is to create a small GPT2 python coder | |
| # Table of Contents | |
| - [Model Card for GPT_2_CODE](#model-card-for--model_id-) | |
| - [Table of Contents](#table-of-contents) | |
| - [Table of Contents](#table-of-contents-1) | |
| - [Model Details](#model-details) | |
| - [Model Description](#model-description) | |
| - [Uses](#uses) | |
| - [Direct Use](#direct-use) | |
| - [Downstream Use [Optional]](#downstream-use-optional) | |
| - [Out-of-Scope Use](#out-of-scope-use) | |
| - [Bias, Risks, and Limitations](#bias-risks-and-limitations) | |
| - [Recommendations](#recommendations) | |
| - [Training Details](#training-details) | |
| - [Training Data](#training-data) | |
| - [Training Procedure](#training-procedure) | |
| - [Preprocessing](#preprocessing) | |
| - [Speeds, Sizes, Times](#speeds-sizes-times) | |
| - [Evaluation](#evaluation) | |
| - [Testing Data, Factors & Metrics](#testing-data-factors--metrics) | |
| - [Testing Data](#testing-data) | |
| - [Factors](#factors) | |
| - [Metrics](#metrics) | |
| - [Results](#results) | |
| - [Model Examination](#model-examination) | |
| - [Environmental Impact](#environmental-impact) | |
| - [Technical Specifications [optional]](#technical-specifications-optional) | |
| - [Model Architecture and Objective](#model-architecture-and-objective) | |
| - [Compute Infrastructure](#compute-infrastructure) | |
| - [Hardware](#hardware) | |
| - [Software](#software) | |
| - [Citation](#citation) | |
| - [Glossary [optional]](#glossary-optional) | |
| - [More Information [optional]](#more-information-optional) | |
| - [Model Card Authors [optional]](#model-card-authors-optional) | |
| - [Model Card Contact](#model-card-contact) | |
| - [How to Get Started with the Model](#how-to-get-started-with-the-model) | |
| # Model Details | |
| ## Model Description | |
| <!-- Provide a longer summary of what this model is/does. --> | |
| WIP, | |
| Goal is to create a small GPT2 python coder | |
| - **Developed by:** C, o, d, e, M, o, n, k, e, y | |
| - **Shared by [Optional]:** More information needed | |
| - **Model type:** Language model | |
| - **Language(s) (NLP):** eng | |
| - **License:** wtfpl | |
| - **Parent Model:** More information needed | |
| - **Resources for more information:** More information needed | |
| - [GitHub Repo](None) | |
| - [Associated Paper](None) | |
| # Uses | |
| <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> | |
| ## Direct Use | |
| <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> | |
| <!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." --> | |
| generate python code snippets | |
| ## Downstream Use [Optional] | |
| <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app --> | |
| <!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." --> | |
| semi auto coder | |
| ## Out-of-Scope Use | |
| <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. --> | |
| <!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." --> | |
| describe code | |
| # Bias, Risks, and Limitations | |
| <!-- This section is meant to convey both technical and sociotechnical limitations. --> | |
| Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. | |
| ## Recommendations | |
| <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> | |
| Keep Finetuning on question/python datasets | |
| # Training Details | |
| ## Training Data | |
| <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> | |
| flytech/python-codes-25k | |
| espejelomar/code_search_net_python_10000_examples | |
| ## Training Procedure | |
| <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> | |
| ### Preprocessing | |
| More information needed | |
| ### Speeds, Sizes, Times | |
| <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> | |
| Epochs 3 | |
| flytech/python-codes-25k (4600) | |
| Training Loss: 0.4007 | |
| Validation Loss: 0.5526 | |
| Epochs 3 | |
| espejelomar/code_search_net_python_10000_examples (4800) | |
| Training Loss: 1.5355 | |
| Validation Loss: 1.1723 | |
| # Evaluation | |
| <!-- This section describes the evaluation protocols and provides the results. --> | |
| ## Testing Data, Factors & Metrics | |
| ### Testing Data | |
| <!-- This should link to a Data Card if possible. --> | |
| flytech/python-codes-25k | |
| espejelomar/code_search_net_python_10000_examples | |
| ### Factors | |
| <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. --> | |
| 80/20 train/val | |
| ### Metrics | |
| <!-- These are the evaluation metrics being used, ideally with a description of why. --> | |
| train/validate | |
| lr scheduling | |
| ## Results | |
| Better in python code generation as base gpt2-medium model | |
| # Model Examination | |
| More information needed | |
| # Environmental Impact | |
| <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> | |
| Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). | |
| - **Hardware Type:** CPU and Colab T4 | |
| - **Hours used:** 4 | |
| - **Cloud Provider:** Google Colab | |
| - **Compute Region:** NL | |
| - **Carbon Emitted:** ??? | |
| # Technical Specifications [optional] | |
| ## Model Architecture and Objective | |
| gpt2 | |
| ## Compute Infrastructure | |
| More information needed | |
| ### Hardware | |
| CPU and Colab T4 | |
| ### Software | |
| pytorch, custom python | |
| # Citation | |
| <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> | |
| **BibTeX:** | |
| More information needed | |
| **APA:** | |
| More information needed | |
| # Glossary [optional] | |
| <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. --> | |
| More information needed | |
| # More Information [optional] | |
| Experimental | |
| # Model Card Authors [optional] | |
| <!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. --> | |
| CodeMonkeyXL | |
| # Model Card Contact | |
| K00B404 huggingface | |
| # How to Get Started with the Model | |
| Use the code below to get started with the model. | |
| <details> | |
| <summary> Click to expand </summary> | |
| More information needed | |
| </details> |