description

src/display/about.py

@@ -28,31 +28,40 @@ TITLE = """<h1 align="center" id="space-title"><code>/\/</code> &nbsp; Open CoT
 INTRODUCTION_TEXT = """
 The `/\/` Open CoT Leaderboard tracks the reasoning skills of LLMs, measured as their ability to generate **effective chain-of-thought reasoning traces**.
 
-The leaderboard reports **accuracy gains** achieved by using CoT, i.e.: _accuracy gain Δ_ = _CoT accuracy_ – _baseline accuracy_.
+The leaderboard reports **accuracy gains** achieved by using CoT, i.e.: _accuracy gain Δ_ = _CoT accuracy_ — _baseline accuracy_.
 
 See the "About" tab for more details and motivation.
 """
 
 # Which evaluations are you running? how can people reproduce what you have?
 LLM_BENCHMARKS_TEXT = """
-## How it works
+## How it works (roughly)
 
 To assess the reasoning skill of a given `model`, we carry out the following steps for each `task` (test dataset) and different CoT `regimes`. (A CoT `regime` consists in a prompt chain and decoding parameters used to generate a reasoning trace.)
 
-1. Let the `model` generate CoT reasoning traces for all problems in the test dataset according to `regime`.
-2. Let the `model` answer the test dataset problems, and record the resulting _baseline accuracy_.
-3. Let the `model` answer the test dataset problems _with the reasoning traces appended_ to the prompt, and record the resulting _CoT accuracy_.
-4. Compute the _accuracy gain Δ_ = _CoT accuracy_ – _baseline accuracy_ for the given `model`, `task`, and `regime`.
+1. `model` generates CoT reasoning traces for all problems in the test dataset according to `regime`.
+2. `model` answers the test dataset problems, we record the resulting _baseline accuracy_.
+3. `model` answers the test dataset problems _with the reasoning traces appended_ to the prompt, we record the resulting _CoT accuracy_.
+4. We compute the _accuracy gain Δ_ = _CoT accuracy_ — _baseline accuracy_ for the given `model`, `task`, and `regime`.
 
-Each `regime` has a different accuracy gain Δ, and the leaderboard reports the best Δ achieved by any regime.
+Each `regime` has a different _accuracy gain Δ_, and the leaderboard reports (for every `model`/`task`) the best Δ achieved by any regime. All models are evaluated with the same set of regimes.
 
 
 ## How is it different from other leaderboards?
 
-...
+Performance leaderboards like the [🤗 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) or [YALL](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard) do a great job in ranking models according task performance.
+
+|🤗 Open LLM Leaderboard |`/\/` Open CoT Leaderboard |
+|---|---|
+|Can `model` solve task?|Does `model` do CoT to improve in task?|
+|Measures absolute performance.|Measures relative performance gains.|
+|Covers broad spectrum of `tasks`.|Focuses on critical thinking `tasks`.|
+
 
 ## Test dataset selection (`tasks`)
 
+The test dataset porblems in the CoT Leaderboard can be solved through clear thinking alone, no specific knowledge is required to do so. They are subsets of the AGIEval benchmark and re-published as `logikon-bench`. The `logiqa` dataset has been newly translated from Chinese to English.
+
 
 ## Reproducibility
 To reproduce our results, check out the repository [cot-eval](https://github.com/logikon-ai/cot-eval).