metadata

title: Edge Llm Leaderboard
emoji: 🌖
colorFrom: red
colorTo: blue
sdk: gradio
sdk_version: 5.8.0
app_file: app.py
pinned: true
license: apache-2.0
tags:
  - edge llm leaderboard
  - llm edge leaderboard
  - llm
  - edge
  - leaderboard

LLM-perf leaderboard

📝 About

The Edge-LLM Leaderboard is a leaderboard to gauge practical performance and quality of edge LLMs. Its aim is to benchmark the performance (throughput and memory) of Large Language Models (LLMs) on Edge hardware - starting with a Raspberry Pi 5 (8GB) based on the ARM Cortex A76 CPU.

Anyone from the community can request a new base model or edge hardware/backend/optimization configuration for automated benchmarking:

Model evaluation requests will be made live soon, in the meantime feel free to email to - arnav[dot]chavan[@]nyunai[dot]com

✍️ Details

To avoid multi-thread discrepencies, all 4 threads are used on the Pi 5.
LLMs are running on a singleton batch with a prompt size of 512 and generating 128 tokens.

All of our throughput benchmarks are ran by this single tool llama-bench using the power of llama.cpp to guarantee reproducibility and consistency.

🏃 How to run locally

To run the Edge-LLM Leaderboard locally on your machine, follow these steps:

1. Clone the Repository

First, clone the repository to your local machine:

git clone https://huggingface.co/spaces/nyunai/edge-llm-leaderboard
cd edge-llm-leaderboard

2. Install the Required Dependencies

Install the necessary Python packages listed in the requirements.txt file: pip install -r requirements.txt

3. Run the Application

You can run the Gradio application in one of the following ways:

Option 1: Using Python python app.py
Option 2: Using Gradio CLI (include hot-reload) gradio app.py

4. Access the Application

Once the application is running, you can access it locally in your web browser at http://127.0.0.1:7860/