README.md

metadata

language:
  - ja
tags:
  - japanese-stablelm
  - causal-lm
pipeline_tag: text-generation
license:
  - other
extra_gated_heading: Access Japanese StableLM Instruct Alpha
extra_gated_description: >-
  This repository is publicly accessible, but you have to accept the conditions
  to access its files and content.
extra_gated_button_content: Access repository
extra_gated_fields:
  Name: text
  Email: text
  Organization: text
  I agree to accept the conditions and share above info with Stability AI: checkbox
extra_gated_prompt: >
  ### JAPANESE STABLELM RESEARCH LICENSE AGREEMENT

  Dated: August 7, 2023


  "Agreement" means the terms and conditions for use, reproduction, distribution
  and modification of the Software Products set forth herein.


  “Documentation” means any specifications, manuals, documentation, and other
  written information provided by Stability AI related to the Software.


  "Licensee" or "you" means you, or your employer or any other person or entity
  (if you are entering into this Agreement on such person’s or entity's behalf),
  of the age required under applicable laws, rules or regulations to provide
  legal consent and that has legal authority to bind your employer or such other
  person or entity if you are entering in this Agreement on their behalf.


  "Stability AI" or "we" means Stability AI Ltd.


  "Software" means, collectively, Stability AI’s proprietary Japanese StableLM
  made available under this Agreement.


  “Software Products” means Software and Documentation.


  By using or distributing any portion or element of the Software Products, you
  agree to be bound by this Agreement.

  - License Rights and Redistribution.
      - Subject to your compliance with this Agreement and the Documentation, Stability AI grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty free and limited license under Stability AI’s intellectual property or other rights owned by Stability AI embodied in the Software Products to reproduce, distribute, and create derivative works of the Software Products for purposes other than commercial or production use.
      - You will not, and will not permit, assist or cause any third party to use, modify, copy, reproduce, create derivative works of, or distribute the Software Products (or any derivative works thereof, works incorporating the Software Products, or any data produced by the Software), in whole or in part, for any commercial or production purposes.
      - If you distribute or make the Software Products, or any derivative works thereof, available to a third party, you shall (i) provide a copy of this Agreement to such third party, and (ii) retain the following attribution notice within a "Notice" text file distributed as a part of such copies: "Japanese StableLM is licensed under the Japanese StableLM Research License, Copyright (c) Stability AI Ltd. All Rights Reserved.”
      - The licenses granted to you under this Agreement are conditioned upon your compliance with the Documentation and this Agreement, including the Acceptable Use Policy below and as may be updated from time to time in the future on stability.ai, which is hereby incorporated by reference into this Agreement.
  - Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE SOFTWARE
  PRODUCTS  AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN "AS IS"
  BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING,
  WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT,
  MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY
  RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE
  SOFTWARE PRODUCTS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE
  SOFTWARE PRODUCTS AND ANY OUTPUT AND RESULTS.

  - Limitation of Liability. IN NO EVENT WILL STABILITY AI OR ITS AFFILIATES BE
  LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE,
  PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST
  PROFITS OR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR
  PUNITIVE DAMAGES, EVEN IF STABILITY AI OR ITS AFFILIATES HAVE BEEN ADVISED OF
  THE POSSIBILITY OF ANY OF THE FOREGOING.

  - Intellectual Property.
      - No trademark licenses are granted under this Agreement, and in connection with the Software Products, neither Stability AI nor Licensee may use any name or mark owned by or associated with the other or any of its affiliates, except as required for reasonable and customary use in describing and redistributing the Software Products.
      - Subject to Stability AI’s ownership of the Software Products and derivatives made by or for Stability AI, with respect to any derivative works and modifications of the Software Products that are made by you, as between you and Stability AI, you are and will be the owner of such derivative works and modifications.
      - If you institute litigation or other proceedings against Stability AI (including a cross-claim or counterclaim in a lawsuit) alleging that the Software Products or associated outputs or results, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted. You will indemnify and hold harmless Stability AI from and against any claim by any third party arising out of or related to your use or distribution of the Software Products in violation of this Agreement.
  - Term and Termination. The term of this Agreement will commence upon your
  acceptance of this Agreement or access to the Software Products and will
  continue in full force and effect until terminated in accordance with the
  terms and conditions herein. Stability AI may terminate this Agreement if you
  are in breach of any term or condition of this Agreement. Upon termination of
  this Agreement, you shall delete and cease use of the Software Products.
  Sections 2-4 shall survive the termination of this Agreement.

  —----------

  ### Japanese StableLM Acceptable Use Policy

  If you access, use, or distribute any Stability AI models, software, or other
  materials (“Stability Technology”) you agree to this Acceptable Use Policy
  (“Policy”).

  We want everyone to use Stability Technology safely and responsibly. You agree
  you will not use, or allow others to use, Stability Technology to:

  - To violate the law or others’ rights (including intellectual property rights
  and the rights of data privacy and protection), nor will you promote,
  contribute to, encourage, facilitate, plan, incite, or further anyone else’s
  violation of the law or others’ rights;

  - To commit, promote, contribute to, facilitate, encourage, plan, incite, or
  further any of the following:
          - Violence or terrorism;
          - Exploitation or harm to children, including the solicitation, creation, acquisition, or dissemination of child exploitative content;
          - Human trafficking, exploitation, and sexual violence;
          - Harassment, abuse, threatening, stalking, or bullying of individuals or groups of individuals;
          - Discrimination in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services on the basis of race, color, caste, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, or genetic information (including family medical history) except as may be required by applicable law (such as the provision of social security benefits solely to people who meet certain age requirements under the law);
          - Creation of malicious code, malware, computer viruses or any activity that could disable, overburden, interfere with or impair the proper working, integrity, operation or appearance of a website or computer system;
  - For purposes of or for the performance of:
          - Fully automated decision-making, including profiling, with respect to an individual or group of individuals which produces legal effects concerning such individual(s) or similarly significantly affects such individual(s);
          - Systematic or automated scraping, mining, extraction, or harvesting of personally identifiable data, or similar activity, from the output of any Stability Technology except with respect to data that you have provided as input to the Stability Technology and which you are legally entitled to process, for so long as you retain such entitlement;
          - Development, improvement, or manufacture of any weapons of mass destruction (such as nuclear, chemical, or biologic weapons), weapons of war (such as missiles or landmines), or any gain of function-related activities with respect to any pathogens;
          - Mission critical applications or systems where best industry practices require fail-safe controls or performance, including operation of nuclear facilities, aircraft navigation, electrical grids, communication systems, water treatment facilities, air traffic control, life support, weapons systems, or emergency locator or other emergency services;
  - To intentionally deceive or mislead others, including use of Japanese
  StableLM related to the following:
      - Generating, promoting, or furthering fraud or the creation or promotion of disinformation;
      - Generating, promoting, or furthering defamatory content, including the creation of defamatory statements, images, or other content;
      - Generating, promoting, or further distributing spam;
      - Impersonating another individual without consent, authorization, or legal right
      - Representing or misleading people into believing that the use of Japanese StableLM or outputs are human-generated;
      - Generating or facilitating false online engagement, including fake reviews and other means of fake online engagement;
      - Generating or facilitating large-scale political advertisements, propaganda, or influence campaigns;
  - Fail to appropriately disclose to end users any known dangers of your AI
  system or misrepresent or mislead with respect to its abilities.

  Nothing in this AUP is intended to prevent or impede any good faith research,
  testing, or evaluation of Japanese StableLM, or publication related to any of
  the foregoing. If you discover any flaws in Japanese StableLM that may be
  harmful to people in any way, we encourage you to notify us and give us a
  chance to remedy such flaws before others can exploit them. If you have
  questions about this AUP, contact us at [email protected].

Japanese-StableLM-Instruct-Alpha-7B

"A parrot able to speak Japanese, ukiyoe, edo period" — Stable Diffusion XL

Model Description

japanese-stablelm-instruct-alpha-7b is a 7B parameter decoder-only language models pre-trained built on top of the Japanese-StableLM-Base-Alpha-7B model and further fine-tuned on various instruction-following datasets.

Usage

First install additional dependencies in requirements.txt:

pip install sentencepiece einops

Then start generating text with japanese-stablelm-instruct-alpha-7b by using the following code snippet:

import torch
from transformers import LlamaTokenizer, AutoModelForCausalLM

tokenizer = LlamaTokenizer.from_pretrained("novelai/nerdstash-tokenizer-v1", additional_special_tokens=['▁▁'])

model = AutoModelForCausalLM.from_pretrained(
    "stabilityai/japanese-stablelm-instruct-alpha-7b",    
    trust_remote_code=True,
)
model.half()
model.eval()

if torch.cuda.is_available():
    model = model.to("cuda")

def build_prompt(user_query, inputs="", sep="\n\n### "):
    sys_msg = "以下は、タスクを説明する指示と、文脈のある入力の組み合わせです。要求を適切に満たす応答を書きなさい。"
    p = sys_msg
    roles = ["指示", "応答"]
    msgs = [": \n" + user_query, ": "]
    if inputs:
        roles.insert(1, "入力")
        msgs.insert(1, ": \n" + inputs)
    for role, msg in zip(roles, msgs):
        p += sep + role + msg
    return p

# this is for reproducibility.
# feel free to change to get different result
seed = 42
torch.manual_seed(seed)

# Infer with prompt without any additional input
user_inputs = {
    "user_query": "VR とはどのようなものですか？",
    "inputs": ""
}
prompt = build_prompt(**user_inputs)

input_ids = tokenizer.encode(
    prompt, 
    add_special_tokens=False, 
    return_tensors="pt"
)

tokens = model.generate(
    input_ids.to(device=model.device),
    max_new_tokens=256,
    temperature=1,
    top_p=0.95,
    do_sample=True,
)

out = tokenizer.decode(tokens[0][input_ids.shape[1]:], skip_special_tokens=True).strip()
print(out)
"""バーチャルリアリティは、現実の世界のように見える仮想世界の 3D 仮想現実のシミュレーションです。これは、ヘッドセットを介して、ユーザーが見たり、聞いたり、体験できるものです。"""

seed = 42
torch.manual_seed(seed)

# Infer with prompt with additional input
user_inputs = {
    "user_query": "VR について、以下の比較対象との違いを箇条書きで教えてください。",
    "inputs": "比較対象: AR"
}
prompt = build_prompt(**user_inputs)

input_ids = tokenizer.encode(
    prompt, 
    add_special_tokens=False, 
    return_tensors="pt"
)

tokens = model.generate(
    input_ids.to(device=model.device),
    max_new_tokens=256,
    temperature=1,
    top_p=0.95,
    do_sample=True,
)

out = tokenizer.decode(tokens[0][input_ids.shape[1]:], skip_special_tokens=True).strip()
print(out)
"""
以下は、VR と AR の比較対象の比較です。
1. VR はユーザーが3D の世界を体験することを可能にし、ユーザーが自分の目で世界を見ることを可能にします。
2. VR は、ユーザーが目の前の環境をより詳細に感じ、より多くのことができるようにすることを可能にします。
3. VR は、ユーザーの感覚を刺激し、拡張することを可能にします。
4. VR は、視覚的、触覚的、および聴覚的な感覚体験を提供するために使用されます。
5. AR は、現実の世界に重ね合わせて、情報を表示し、ユーザーに拡張現実体験を提供することを可能にします。
6. AR は、ユーザーが仮想オブジェクトを仮想環境に持ち込むことを可能にするため、物理的な世界をシミュレートするのに最適です。
7. VR は、3D 世界を実現する仮想世界を作成することに最適です。
8. AR は、ユーザーが現実世界のオブジェクトをシミュレートし、現実世界の現実的な世界に重ね合わせて情報を表示することを可能にします。
9. VR は、ユーザーの感覚や感情に与える影響が最も大きいと考えられています。
"""

Model Details

• Model type: japanese-stablelm-instruct-alpha-7b is an auto-regressive language model based on the NeoX transformer architecture.
• Language(s): Japanese
• Library: GPT-NeoX
• License: This model is licensed under JAPANESE STABLELM RESEARCH LICENSE AGREEMENT.

Training

Parameters	Hidden Size	Layers	Heads	Sequence Length
7B	4096	32	32	1024

Training Dataset

japanese-stablelm-instruct-alpha-7b is fine-tuned on a combination of following datasets:

• Japanese translation of the Stanfored Alpaca dataset
• Japanese translation of the Databricks Dolly-15k dataset
• Japanese translation of the subset of the Anthropic HH dataset
• Wikinews subset of the izumi-lab/llm-japanese-dataset

Use and Limitations

Intended Use

This model is intended to be used by the open-source community in chat-like applications in adherence with the research license.

Limitations and bias

Although the aforementioned datasets help to steer the base language models into "safer" distributions of text, not all biases and toxicity can be mitigated through fine-tuning. We ask that users be mindful of such potential issues that can arise in generated responses. Do not treat model outputs as substitutes for human judgment or as sources of truth. Please use responsibly.

Authors

• Meng Lee
• Fujiki Nakamura
• Makoto Shing
• Paul McCann
• Takuya Akiba
• Naoki Orii

Acknowledgements

We are utilizing the v1 version of the novelai-tokenizer, introduced by NovelAI, because it processes both Japanese and English text both effectively and efficiently. We extend our gratitude to NovelAI for allowing us to use their remarkable work. For more details about the tokenizer, please refer to their blog post.

We are grateful for the contributions of the EleutherAI Polyglot-JA team in helping us to collect a large amount of pre-training data in Japanese. Polyglot-JA members includes Hyunwoong Ko (Project Lead), Fujiki Nakamura (originally started this project when he committed to the Polyglot team), Yunho Mo, Minji Jung, KeunSeok Im, and Su-Kyeong Jang.

We are also appreciative of AI Novelist/Sta (Bit192, Inc.) and the numerous contributors from Stable Community Japan for assisting us in gathering a large amount of high-quality Japanese textual data for model training.

How to cite

@misc{JapaneseStableLMInstructAlpha7B, 
      url={[https://huggingface.co/stabilityai/japanese-stablelm-instruct-alpha-7b](https://huggingface.co/stabilityai/japanese-stablelm-instruct-alpha-7b)}, 
      title={Japanese StableLM Instruct Alpha 7B}, 
      author={Lee, Meng and Nakamura, Fujiki and Shing, Makoto and McCann, Paul and Akiba, Takuya and Orii, Naoki}
}

Citations

@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}

@software{gpt-neox-library,
  title = {{GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch}},
  author = {Andonian, Alex and Anthony, Quentin and Biderman, Stella and Black, Sid and Gali, Preetham and Gao, Leo and Hallahan, Eric and Levy-Kramer, Josh and Leahy, Connor and Nestler, Lucas and Parker, Kip and Pieler, Michael and Purohit, Shivanshu and Songz, Tri and Phil, Wang and Weinbach, Samuel},
  url = {https://www.github.com/eleutherai/gpt-neox},
  doi = {10.5281/zenodo.5879544},
  month = {8},
  year = {2021},
  version = {0.0.1},
}