feat: add eval doc

Former-commit-id: 8a1e2a05e7bbb21b994c65e9cd95cad2e4cf8e72 [formerly 0af2bc5586b798405e4aa7f1ec18e7faba86ab97]
Former-commit-id: 6fbc84bd2c4694f132d1e0958e768f9c4e3a9eb5

reports/eval.qmd

@@ -0,0 +1,44 @@
+---
+title: Evaluation of Summaries
+author: Cillian Berragan
+format: pdf
+fontfamily: libertinus
+monofont: 'JetBrains Mono'
+monofontoptions:
+  - Scale=0.75
+---
+
+This document compares summaries written by Cambridge, to the summaries generated automatically by our model.
+
+# Overview
+
+For all representations, the original summary was compared with the generated summary provided by the LLM. A separate LLM call was used to determine which of these two summaries was preferred, based on set criteria:
+
+> A good summary should:  
+> 1. **Be accurate** – It should not include information that is not present in the source document.  
+> 2. **Be comprehensive** – It should reflect all key points in the source document without omitting important details.  
+> 3. **Be well-grounded** – It should be based entirely on the source document without adding interpretations, opinions, or external information.  
+
+This model was given the option to return 4 different scores; 0 meaning neither summaries are suitable, 1 meaning the original summary is preferred, 2 meaning the LLM-generated summary is preferred, or 3 meaning both summaries are suitable.
+
+@tbl-eval gives the results of this processing. We can see that the majority of the preferred summaries are those generated by the LLM (2). There are however 8 cases where the original summary is considered better, and 17 where both summaries are considered suitable.
+
+```{python}
+#| label: tbl-eval
+#| caption: Comparison between original summary and LLM-generated summary
+#| echo: false
+#| output: asis
+
+import polars as pl
+
+summaries = pl.read_parquet("./data/out/eval.parquet")
+print(
+    summaries["score"]
+    .value_counts()
+    .sort("score")
+    .transpose(include_header=True)
+    .rename({"column_0": "Original", "column_1": "LLM-generated", "column_2": "Both"})
+    .drop("column").tail(1)
+    .to_pandas().to_markdown(index=False)
+)
+```