You need to agree to share your contact information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
STABILITY AI NON-COMMERCIAL RESEARCH COMMUNITY LICENSE AGREEMENT
Dated: November 20, 2024
By clicking “I Accept” below or by using or distributing any portion or element of the Models, Software, Software Products or Derivative Works, you agree to the terms of this License. If you do not agree to this License, then you do not have any rights to use the Software Products or Derivative Works through this License, and you must immediately cease using the Software Products or Derivative Works. If you are agreeing to be bound by the terms of this License on behalf of your employer or other entity, you represent and warrant to Stability AI that you have full legal authority to bind your employer or such entity to this License. If you do not have the requisite authority, you may not accept the License or access the Software Products or Derivative Works on behalf of your employer or other entity.
“Agreement“ means this Stable Non-Commercial Research Community License Agreement.
“AUP” means the Stability AI Acceptable Use Policy available at https://stability.ai/use-policy, as may be updated from time to time.
“Derivative Work(s)” means (a) any derivative work of the Software Products as recognized by U.S. copyright laws, (b) any modifications to a Model, and (c) any other model created which is based on or derived from the Model or the Model’s output. For clarity, Derivative Works do not include the output of any Model.
“Documentation” means any specifications, manuals, documentation, and other written information provided by Stability AI related to the Software.
“Licensee“ or “you“ means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity's behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
“Model(s)“ means, collectively, Stability AI’s proprietary models and algorithms, including machine-learning models, trained model weights and other elements of the foregoing, made available under this Agreement.
“Non-Commercial Uses” means exercising any of the rights granted herein for the purpose of research or non-commercial purposes. Non-Commercial Uses does not include any production use of the Software Products or any Derivative Works.
“Stability AI“ or “we“ means Stability AI Ltd and its affiliates.
“Software“ means Stability AI’s proprietary software made available under this Agreement.
“Software Products” means the Models, Software and Documentation, individually or in any combination.
1. License Rights and Redistribution.
a. Subject to your compliance with this Agreement, the AUP (which is hereby incorporated herein by reference), and the Documentation, Stability AI grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty free and limited license under Stability AI’s intellectual property or other rights owned or controlled by Stability AI embodied in the Software Products to use, reproduce, distribute, and create Derivative Works of, the Software Products, in each case for Non-Commercial Uses only.
b. You may not use the Software Products or Derivative Works to enable third parties to use the Software Products or Derivative Works as part of your hosted service or via your APIs, whether you are adding substantial additional functionality thereto or not. Merely distributing the Software Products or Derivative Works for download online without offering any related service (ex. by distributing the Models on HuggingFace) is not a violation of this subsection. If you wish to use the Software Products or any Derivative Works for commercial or production use or you wish to make the Software Products or any Derivative Works available to third parties via your hosted service or your APIs, contact Stability AI at https://stability.ai/contact.
c. If you distribute or make the Software Products, or any Derivative Works thereof, available to a third party, the Software Products, Derivative Works, or any portion thereof, respectively, will remain subject to this Agreement and you must (i) provide a copy of this Agreement to such third party, and (ii) retain the following attribution notice within a “Notice“ text file distributed as a part of such copies: “This Stability AI Model is licensed under the Stability AI Non-Commercial Research Community License, Copyright (c) Stability AI Ltd. All Rights Reserved.“ If you create a Derivative Work of a Software Product, you may add your own attribution notices to the Notice file included with the Software Product, provided that you clearly indicate which attributions apply to the Software Product and you must state in the NOTICE file that you changed the Software Product and how it was modified.
2. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE SOFTWARE PRODUCTS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS“ BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE SOFTWARE PRODUCTS, DERIVATIVE WORKS OR ANY OUTPUT OR RESULTS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE SOFTWARE PRODUCTS, DERIVATIVE WORKS AND ANY OUTPUT AND RESULTS.
3. Limitation of Liability. IN NO EVENT WILL STABILITY AI OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN IF STABILITY AI OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
4. Intellectual Property.
a. No trademark licenses are granted under this Agreement, and in connection with the Software Products or Derivative Works, neither Stability AI nor Licensee may use any name or mark owned by or associated with the other or any of its affiliates, except as required for reasonable and customary use in describing and redistributing the Software Products or Derivative Works.
b. Subject to Stability AI’s ownership of the Software Products and Derivative Works made by or for Stability AI, with respect to any Derivative Works that are made by you, as between you and Stability AI, you are and will be the owner of such Derivative Works.
c. If you institute litigation or other proceedings against Stability AI (including a cross-claim or counterclaim in a lawsuit) alleging that the Software Products, Derivative Works or associated outputs or results, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted. You will indemnify and hold harmless Stability AI from and against any claim by any third party arising out of or related to your use or distribution of the Software Products or Derivative Works in violation of this Agreement.
5. Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access to the Software Products and will continue in full force and effect until terminated in accordance with the terms and conditions herein. Stability AI may terminate this Agreement if you are in breach of any term or condition of this Agreement. Upon termination of this Agreement, you shall delete and cease use of any Software Products or Derivative Works. Sections 2-4 shall survive the termination of this Agreement.
6. Governing Law. This Agreement will be governed by and construed in accordance with the laws of the United States and the State of California without regard to choice of law principles.
Please learn more about our self-hosted Membership options here.
Log in or Sign Up to review the conditions and access this model content.
Arabic Stable LM 2 1.6B
Please note: For commercial use, please refer to https://stability.ai/license
Model Description
Arabic Stable LM 2 1.6B is a fine-tuned model from Stable LM2 1.6B that has been fine-tuned on more than 100 Billion tokens of Arabic text.
Usage
Get started generating text with Stable LM 2 1.6B by using the following code snippet:
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("stabilityai/ar-stablelm-2-base")
model = AutoModelForCausalLM.from_pretrained(
"stabilityai/ar-stablelm-2-base",
torch_dtype="auto",
)
model.cuda()
inputs = tokenizer("يعتبر عيد الأضحى", return_tensors="pt").to(model.device)
tokens = model.generate(
**inputs,
max_new_tokens=64,
temperature=0.70,
top_p=0.95,
do_sample=True,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
Run with Flash Attention 2 ⚡️
Click to expand
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("stabilityai/ar-stablelm-2-base")
model = AutoModelForCausalLM.from_pretrained(
"stabilityai/ar-stablelm-2-base",
torch_dtype="auto",
attn_implementation="flash_attention_2",
)
model.cuda()
inputs = tokenizer("يعتبر عيد الأضحى", return_tensors="pt").to(model.device)
tokens = model.generate(
**inputs,
max_new_tokens=64,
temperature=0.70,
top_p=0.95,
do_sample=True,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
Model Details
- Developed by: Stability AI
- Model type:
Arabic Stable LM 2 1.6Bmodels are auto-regressive language models based on the transformer decoder architecture. - Language(s): Arabic
- Paper: Arabic Stable LM 2 1.6B paper
- Library: GPT-NeoX
- License:Stability AI Non-Commercial Research Community License
- Contact: Please contact us:
- Questions and comments about the model: [email protected]
- Safety issues: [email protected]
- Security issues: [email protected]
- Privacy issues: [email protected]
Model Architecture
The model is a decoder-only transformer similar to the LLaMA (Touvron et al., 2023) architecture with the following modifications:
| Parameters | Hidden Size | Layers | Heads | Sequence Length |
|---|---|---|---|---|
| 1,644,417,024 | 2048 | 24 | 32 | 4096 |
- Position Embeddings: Rotary Position Embeddings (Su et al., 2021) applied to the first 25% of head embedding dimensions for improved throughput following Black et al. (2022).
- Normalization: LayerNorm (Ba et al., 2016) with learned bias terms as opposed to RMSNorm (Zhang & Sennrich, 2019).
- Biases: We remove all bias terms from the feed-forward networks and multi-head self-attention layers, except for the biases of the query, key, and value projections (Bai et al., 2023).
- Tokenizer: We use Arcade100k, a BPE tokenizer extended from OpenAI's
tiktoken.cl100k_base. We split digits into individual tokens following findings by Liu & Low (2023).
Training
Training Dataset
The model is trained on a mixture of English and Arabic datasets where 619 billion tokens for English and around 115 billion tokens for Arabic.
- Fine-tuning the base
Arabic Stable LM 2 1.6Bfor the user’s downstream tasks is recommended.
Training Procedure
The model is a fine-tuned version of Stable LM 1.6B model using a learning scheduler with early cool down. The model is fine-tuned for 300k steps using a cosine and inverse square-root and 200k using a cool down with linear learning rate.
Training Infrastructure
Hardware: We use two nodes for the training, each with 8 H100 GPUs with a micro-batch size of 6 per GPU. This results in a global batch size of 6 × 2 × 8 = 96 sequences that sums up to around 400K tokens per batch. The full training setup with 500k steps consumes around 197B tokens.
Software: We use a fork of
gpt-neox(EleutherAI, 2021), train under 2D parallelism (Data and Tensor Parallel) with ZeRO-1 (Rajbhandari et al., 2019), and rely on flash-attention as well as SwiGLU and Rotary Embedding kernels from FlashAttention-2 (Dao et al., 2023)
Use and Limitations
Intended Use
The model is intended to be used as a foundational base model for application-specific fine-tuning for research only. Users should evaluate the model for safety performance in their specific use case and apply the necessary safeguards and fine-tune the model to facilitate safe performance in downstream applications.
Out-of-scope Use
Out-of-scope uses include use in any manner that violates applicable laws or regulations, Stability AI’s Acceptable Use Policy or license agreement, or use in languages outside of those explicitly supported by this model.
Limitations and Bias
As a base model, this model may exhibit unreliable or other undesirable behaviors that should be corrected through evaluation and fine-tuning prior to deployment. Given that each use case is unique, running a suite of tests may help facilitate proper performance of this model. Using this model will require guardrails around the user’s inputs and outputs to ensure that any outputs returned are not harmful. Pairing this model with an input and output classifier may help prevent harmful responses. Users should exercise caution when using these models in production systems and should not use the models if they are unsuitable for the user’s application.
How to Cite
@misc{alyafeai2024arabicstablelmadapting,
title={Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic},
author={Zaid Alyafeai and Michael Pieler and Hannah Teufel and Jonathan Tow and Marco Bellagente and Duy Phung and Nikhil Pinnaparaju and Reshinth Adithyan and Paulo Rocha and Maksym Zhuravinskyi and Carlos Riquelme},
year={2024},
eprint={2412.04277},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.04277},
}
- Downloads last month
- 86