Hugging Face's logo

qualcomm
/
Llama-v3-8B-Chat

Text Generation
PyTorch
llm
generative_ai
quantized
android
Model card Files Community
2
Llama-v3-8B-Chat / README.md
korywat's picture
korywat
Upload README.md with huggingface_hub
21cb7b6 verified
preview code
|
raw
history blame
6.23 kB
metadata
library_name: pytorch
license: llama3
pipeline_tag: text-generation
tags:
  - llm
  - generative_ai
  - quantized
  - android

Llama-v3-8B-Chat: Optimized for Mobile Deployment

State-of-the-art large language model useful on a variety of language understanding and generation tasks

Llama 3 is a family of LLMs. The "Chat" at the end indicates that the model is optimized for chatbot-like dialogue. The model is quantized to w4a16 (4-bit weights and 16-bit activations) and part of the model is quantized to w8a16 (8-bit weights and 16-bit activations) making it suitable for on-device deployment. For Prompt and output length specified below, the time to first token is Llama-PromptProcessor-Quantized's latency and average time per addition token is Llama-TokenGenerator-Quantized's latency.

This model is an implementation of Llama-v3-8B-Chat found here. This repository provides scripts to run Llama-v3-8B-Chat on Qualcomm® devices. More details on model performance across various devices, can be found here.

Model Details

  • Model Type: Text generation
  • Model Stats:
    • Context length: 4096
    • Number of parameters: 8B
    • Model size: 4.8GB
    • Precision: w4a16 + w8a16 (few layers)
    • Num of key-value heads: 8
    • Model-1 (Prompt Processor): Llama-PromptProcessor-Quantized
    • Prompt processor input: 128 tokens + position embeddings + attention mask + KV cache inputs
    • Prompt processor output: 128 output tokens + KV cache outputs
    • Model-2 (Token Generator): Llama-TokenGenerator-Quantized
    • Token generator input: 1 input token + position embeddings + attention mask + KV cache inputs
    • Token generator output: 1 output token + KV cache outputs
    • Use: Initiate conversation with prompt-processor and then token generator for subsequent iterations.

Deploying Llama 3 on-device

Please follow this tutorial to compile QNN binaries and generate bundle assets to run ChatApp on Windows and on Android powered by QNN-Genie.

Sample output prompts generated on-device

  1. --prompt "where is California?"
------- Response Summary --------
Prompt: where is California?
Response: California is a state located on the West Coast of
  1. --prompt "what is 2+3?" --max-output-tokens 30
-------- Response Summary --------
Prompt: what is 2+3?
Response: 2 + 3 = 5
  1. --prompt "what is superposition in Quantum Physics?" --max-output-tokens 30
Prompt: what is superposition in Quantum Physics?
Response: Superposition is a fundamental concept in quantum mechanics, which is a branch of physics that studies the behavior of matter and energy at a very
Device Chipset Target Runtime Inference Time (ms) Peak Memory Range (MB) Precision Primary Compute Unit Target Model

Installation

This model can be installed as a Python package via pip.

pip install "qai-hub-models[llama_v3_8b_chat_quantized]"

Configure Qualcomm® AI Hub to run this model on a cloud-hosted device

Sign-in to Qualcomm® AI Hub with your Qualcomm® ID. Once signed in navigate to Account -> Settings -> API Token.

With this API token, you can configure your client to run models on the cloud hosted devices.

qai-hub configure --api_token API_TOKEN

Navigate to docs for more information.

Demo off target

The package contains a simple end-to-end demo that downloads pre-trained weights and runs this model on a sample input.

python -m qai_hub_models.models.llama_v3_8b_chat_quantized.demo

The above demo runs a reference implementation of pre-processing, model inference, and post processing.

NOTE: If you want running in a Jupyter Notebook or Google Colab like environment, please add the following to your cell (instead of the above).

%run -m qai_hub_models.models.llama_v3_8b_chat_quantized.demo

Run model on a cloud-hosted device

In addition to the demo, you can also run the model on a cloud-hosted Qualcomm® device. This script does the following:

  • Performance check on-device on a cloud-hosted device
  • Downloads compiled assets that can be deployed on-device for Android.
  • Accuracy check between PyTorch and on-device outputs.
python -m qai_hub_models.models.llama_v3_8b_chat_quantized.export

Deploying compiled model to Android

The models can be deployed using multiple runtimes:

  • TensorFlow Lite (.tflite export): This tutorial provides a guide to deploy the .tflite model in an Android application.

  • QNN (.so export ): This sample app provides instructions on how to use the .so shared library in an Android application.

View on Qualcomm® AI Hub

Get more details on Llama-v3-8B-Chat's performance across various devices here. Explore all available models on Qualcomm® AI Hub

License

  • The license for the original implementation of Llama-v3-8B-Chat can be found here.
  • The license for the compiled assets for on-device deployment can be found here

References

Community