update about page text

src/about.py

@@ -9,21 +9,53 @@ INTRODUCTION_TEXT = """
 """
 
 # Which evaluations are you running? how can people reproduce what you have?
+# LLM_BENCHMARKS_TEXT = """
+# 1) GPT-4T was used except for some accuracy use cases with atypically long input tokens.
+# 2) Hyperparameters were optimized for a subset of models evaluated (platform models?) Were parameters optimized as well?
+# 3) Latency reflects the mean latency over a single time range on a high-speed internet connection; response times for external APIs may vary over time and be impacted by internet speed, location, etc.
+# 3) Latency reflects the time to receive the entire completion.
+# 4) Some external APIs were direct to the LLM provider (OpenAI, Google, AI21), while others were provided through Amazon Bedrock (Cohere, Anthropic)
+# 5) LLM annotations (manual/human evaluations) were performed under a variety of settings that did not necessarily control for ordering effects
+# 6) All tests on open source models were performed on original models (correct?); custom fine-tuning may impact performance in trust / safety / toxicity / bias / etc.
+# 7) For the tests on latency, the inputs were *approximately* 500 / 3000 tokens. A short prompt was added and different models tokenize differently.
+# 8) Costs for all external APIs were based on the standard pricing of the provider (note that the pricing of cohere/anthropic via Bedrock is the same as directly through Cohere/Anthropic apis).
+# 9) Something about limitations of LLM judges (despite correlation with human annotators)
+# 10) Task-specific model variants were not used from the external providers (command-r is sort of retrieval specific, but this was not one of the use cases)
+# 11) Maybe something about the tasks being primarily summarization / generation
+# 12) CRM T&S is done by perturbing words: 1) for gender bias, we perturb person names and pronouns to opposite gender. 2) for entity bias, we perturb company names to its competitors in the same sector
+# 13) Cost per request for self-hosted models assume a minimal frequency of calling the model, since the costs are per hour. All latencies / cost assume a single user at a time.
+# """
+
 LLM_BENCHMARKS_TEXT = """
-1) GPT-4T was used except for some accuracy use cases with atypically long input tokens.
-2) Hyperparameters were optimized for a subset of models evaluated (platform models?) Were parameters optimized as well?
-3) Latency reflects the mean latency over a single time range on a high-speed internet connection; response times for external APIs may vary over time and be impacted by internet speed, location, etc.
+0) We consider models that are general instruction-tuned, not task-specific fine-tuned ones
+1) For GPT-4/GPT-4-Turbo Models, following tasks were evaluated:
+  + GPT4:
+    - Service: Conversation summary
+    - Sales: Email Generation
+    - Sales & Service: Update CRM Info
+    - Service: Reply Recommendations
+
+  + GPT4-Turbo:
+    - Service: Live Chat Insights
+    - Service: Email Summary
+    - Sales: Call Summary
+    - Service: Knowledge creation from Case Info
+    - Service: Call Summary
+    - Service: Live Chat Summary
+
+2) Latency reflects the mean latency over a single time range on a high-speed internet connection; response times for external APIs may vary over time and be impacted by internet speed, location, etc.
 3) Latency reflects the time to receive the entire completion.
 4) Some external APIs were direct to the LLM provider (OpenAI, Google, AI21), while others were provided through Amazon Bedrock (Cohere, Anthropic)
 5) LLM annotations (manual/human evaluations) were performed under a variety of settings that did not necessarily control for ordering effects
 6) All tests on open source models were performed on original models (correct?); custom fine-tuning may impact performance in trust / safety / toxicity / bias / etc.
 7) For the tests on latency, the inputs were *approximately* 500 / 3000 tokens. A short prompt was added and different models tokenize differently.
 8) Costs for all external APIs were based on the standard pricing of the provider (note that the pricing of cohere/anthropic via Bedrock is the same as directly through Cohere/Anthropic apis).
-9) Something about limitations of LLM judges (despite correlation with human annotators)
-10) Task-specific model variants were not used from the external providers (command-r is sort of retrieval specific, but this was not one of the use cases)
+9) Building a reliable LLM-based evaluator remains an open challenge due to inherent biases such as 1) Length Bias: the tendency to favor longer responses, and 2) Self-enhancement Bias: the tendency of the LLM-evaluator to favor its own responses.
+10) Task-specific model variants were not used from the external providers (command-r is sort of retrieval specific, but this was not one of the use cases).
 11) Maybe something about the tasks being primarily summarization / generation
-12) CRM T&S is done by perturbing words: 1) for gender bias, we perturb person names and pronouns to opposite gender. 2) for entity bias, we perturb company names to its competitors in the same sector
+12) Trust & Safety was benchmarked on public datasets as well as bias perturbations on CRM datasets. For gender bias, person names and pronouns were perturbed. For company bias, company names were perturbed to competitors in the same sector. For the CRM Bias metric, higher means less bias.
 13) Cost per request for self-hosted models assume a minimal frequency of calling the model, since the costs are per hour. All latencies / cost assume a single user at a time.
+14) The current auto-evaluation is based on LLaMA-70B as Judge, which showed the highest correlation with human annotaotors.
 """
 
 CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"