|
--- |
|
title: Model Evaluator |
|
emoji: π |
|
colorFrom: red |
|
colorTo: red |
|
sdk: streamlit |
|
sdk_version: 1.10.0 |
|
app_file: app.py |
|
--- |
|
|
|
# Model Evaluator |
|
|
|
> Submit evaluation jobs to AutoTrain from the Hugging Face Hub |
|
|
|
## Supported tasks |
|
|
|
The table below shows which tasks are currently supported for evaluation in the AutoTrain backend: |
|
|
|
| Task | Supported | |
|
|:-----------------------------------|:---------:| |
|
| `binary_classification` | β
| |
|
| `multi_class_classification` | β
| |
|
| `multi_label_classification` | β | |
|
| `entity_extraction` | β
| |
|
| `extractive_question_answering` | β
| |
|
| `translation` | β
| |
|
| `summarization` | β
| |
|
| `image_binary_classification` | β
| |
|
| `image_multi_class_classification` | β
| |
|
|
|
## Installation |
|
|
|
To run the application locally, first clone this repository and install the dependencies as follows: |
|
|
|
``` |
|
pip install -r requirements.txt |
|
``` |
|
|
|
Next, copy the example file of environment variables: |
|
|
|
``` |
|
cp .env.template .env |
|
``` |
|
|
|
and set the `HF_TOKEN` variable with a valid API token from the `autoevaluator` user. Finally, spin up the application by running: |
|
|
|
``` |
|
streamlit run app.py |
|
``` |
|
|
|
## AutoTrain configuration details |
|
|
|
Models are evaluated by AutoTrain, with the payload sent to the `AUTOTRAIN_BACKEND_API` environment variable. The current configuration for evaluation jobs running on Spaces is: |
|
|
|
``` |
|
AUTOTRAIN_BACKEND_API=https://api-staging.autotrain.huggingface.co |
|
``` |
|
|
|
To evaluate models with a _local_ instance of AutoTrain, change the environment to: |
|
|
|
``` |
|
AUTOTRAIN_BACKEND_API=http://localhost:8000 |
|
``` |