Spaces:
Running
Running
| # Your leaderboard name | |
| TITLE = """<h1 align="center" id="space-title">LLM Leaderboard for CRM</h1> | |
| <h3>Assess which LLMs are accurate enough or need fine-tuning, and weigh this versus tradeoffs of speed, costs, and trust and safety. This is based on human manual and automated evaluation with real operational CRM data per use case.</h3> | |
| """ | |
| # What does your leaderboard evaluate? | |
| INTRODUCTION_TEXT = """ | |
| """ | |
| LLM_BENCHMARKS_TEXT = """ | |
| 1) We consider models that are general instruction-tuned, not task-specific fine-tuned ones | |
| 2) For GPT-4/GPT-4-Turbo Models, following tasks were evaluated: | |
| + GPT4: | |
| - Service: Conversation summary | |
| - Sales: Email Generation | |
| - Sales & Service: Update CRM Info | |
| - Service: Reply Recommendations | |
| + GPT4-Turbo: | |
| - Service: Live Chat Insights | |
| - Service: Email Summary | |
| - Sales: Call Summary | |
| - Service: Knowledge creation from Case Info | |
| - Service: Call Summary | |
| - Service: Live Chat Summary | |
| 3) Latency scores reflect the mean latency on a high-speed internet connection over a particular time span, based on the time to receive the entire completion; response times for external APIs may vary and be impacted by internet speed, location, etc. | |
| 4) Some external APIs were hosted directly by the LLM provider (OpenAI, Google, AI21), while others were provided through Amazon Bedrock (Cohere, Anthropic) | |
| 5) LLM annotations (manual/human evaluations) were performed under a variety of settings that did not necessarily control for ordering effects. | |
| 6) For the tests on latency: two cases were considered: (1) Length ~500 input and length ~250 output, and (2) length ~3000 input and ~250 output, reflecting common use cases for summarization and generation tasks. | |
| 7) Costs for all external APIs were based on the standard pricing of the provider (note that the pricing of cohere/anthropic via Bedrock is the same as directly through Cohere/Anthropic APIs). | |
| 8) Trust & Safety was benchmarked on public datasets as well as bias perturbations on CRM datasets. For gender bias, person names and pronouns were perturbed. For company bias, company names were perturbed to competitors in the same sector. For the CRM Fairness metric, higher means less bias. | |
| 9) Cost per request for self-hosted models assume a minimal frequency of calling the model, since the costs are per hour. All latencies / cost assume a single user at a time. | |
| 10) The current auto-evaluation is based on LLaMA-70B as Judge, which showed the highest correlation with human annotators; however, LLM judges may be less reliable than human annotators. This remains an active area of research. | |
| """ | |
| CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results" | |
| CITATION_BUTTON_TEXT = r""" | |
| @misc{crm-llm-leaderboard, | |
| author = {Salesforce AI}, | |
| title = {LLM Leaderboard for CRM}, | |
| year = {2024}, | |
| publisher = {Salesforce AI}, | |
| howpublished = "\url{https://https://huggingface.co/spaces/Salesforce/crm_llm_leaderboard}" | |
| } | |
| """ | |