add reports dir

reports/DOCS/DOCS.qmd

@@ -0,0 +1,98 @@
+---
+title: 'Summarisation of Planning Responses with LLMs'
+format:
+  PrettyPDF-pdf:
+    papersize: A4
+execute:
+  freeze: auto
+  echo: false
+monofont: 'JetBrains Mono'
+monofontoptions:
+  - Scale=0.55
+---
+
+## Introduction
+
+* Saves time; takes minutes rather than hours (or days)
+* Reduces bias?
+* All information can be considered equally.
+* Diverse forms of input; from letters to brief comments, even handwritten text (gpt-4o).
+* Generate easy to understand summaries, removing any domain specific terminology; increased transparency etc.
+
+* Need to ensure the summaries are accurate, may require human oversight? For this we are using hallucination detection which works well. Eval vs summaries already generated?
+
+## Methodology
+
+This project primarily considers the use of generative pre-trained transformer (GPT) large-language models (LLMs) for _abstractive_ summarisation of planning responses. Unlike _extractive_ summarisation, where encoder-transformer LLMs has been an established task for a number of years (e.g. with Google's BERT), this task has now been advanced through the use of larger scale GPT models (e.g. OpenAIs gpt-3/gpt-4 series). One benefit of these new models are their size; they are both _trained_ on more human data, and have a larger number of _model parameters_. Both of these factors mean that such models are able to understand human text and semantic nuances to a greater degree. Subsequently, their architectural differences mean that, while BERT-like models excel at _extractive summarisation_, GPT models are able to _generate_ large amounts of human-like text. 
+
+Given these advances, a number of methods relating to document summarisation have been established in recent years (and months). In this project, we focus on the task of _map-reduce_ summarisation; given a large set of documents, summarise each, then summarise those summaries to produce a final report.
+
+For our use-case we established the following data-flow;
+
+NOTE: I want some way to integrate 'citations'. One approach is to use extractive summarisation to show related passages? Could also use inline citations, but there can be quite a few of those (700?). Maybe a way to reduce them down, splitting to sentences, and grouping?
+
+```{mermaid}
+%%{init: {'flowchart': {'curve': 'linear'}}}%%
+graph TD;
+        __start__([__start__]):::first
+        generate_summary(generate_summary)
+        check_hallucination(check_hallucination)
+        fix_hallucination(fix_hallucination)
+        generate_final_summary(generate_final_summary)
+        __end__([__end__]):::last
+        check_hallucination --> generate_final_summary;
+        generate_final_summary --> __end__;
+        __start__ -.-> generate_summary;
+        generate_summary -.-> check_hallucination;
+        check_hallucination -.-> fix_hallucination;
+        fix_hallucination -.-> check_hallucination;
+        classDef default fill:#f2f0ff,line-height:1.2
+        classDef first fill-opacity:0
+        classDef last fill:#bfb6fc
+```
+
+1. Summaries for each response are generated in parallel
+2. Each summary is check to ensure there are no _hallucinations_ (cyclically)  
+3. Summaries are used to form a final report
+
+### Hallucination removal
+
+Hallucination detection and removal is a key part of this process. While summarisation typically accurately reflect the information contained within the documents they are summarising, the model may sometimes inject information that isn't explicitly stated. To avoid these cases we use another LLM _agent_ to read through both the original document and the summarisation, to produce a report highlighting any issues, and give a score that informs a downstream stage whether the summary is accurate.
+
+In the following example we highlight a case where the original summarisation LLM makes a clear mistake, which is then rectified by a further iteration, using information generated by the hallucination agent. For brevity, the questions have been removed from the response.
+
+**Document**:
+
+_None. The wastewater treatment plant has recently been upgraded and deemed fit for purpose for a significant
+ number of years going forward. The carbon and financial cost of relocating this site is huge - if housing is needed then the housing should be situated in the proposed site for the relocated treatment plant, the new
+ homeowners would certainly enjoy their life in the greenbelt, those living nearby would be grateful and £227 million pounds and many tonnes of carbon saved._
+
+_Vast open spaces should be employed. Community centers are important, including support for children and mental health. National chains should be banned from owning shops or property in the area._
+
+_Cambourne should remain isolated and become self sufficient. If anything, more cycle only routes should be set up._
+
+_None. These villages should remain as they are._
+
+_Grantchester._
+
+_Road bypasses._
+
+_Barton and Newnham_
+
+_Underdeveloped and preserved_
+
+**Summarisation Attempt**: 
+
+The response indicates that the area east of Milton Road can be developed into a lively city district after the wastewater treatment plant relocates, emphasizing the high costs of relocation. It advocates for vast open spaces and community centers around the Cambridge Biomedical Campus, while suggesting that Cambourne should remain isolated and self-sufficient. The response opposes development in the southern rural cluster of villages, supports limited development in Grantchester, and proposes road bypasses. Additionally, it identifies Barton and Newnham as potential sites for development. The overall vision for Greater Cambridge in 2041 includes a focus on preservation and limited development.
+
+**Hallucination agent response**
+
+The Assistant's Summary contains several inaccuracies and misinterpretations of the source document. Firstly, it states that the response opposes development east of Milton Road due to the recent upgrade of the wastewater treatment plant, which is misleading. The source document suggests that the area can be developed after the plant relocates, indicating a potential for development rather than outright opposition. Secondly, the summary claims that the vision for Greater Cambridge in 2041 is to remain underdeveloped and preserved, which is a misrepresentation of the nuanced views expressed in the source document. The source document does advocate for some development in specific areas, such as Grantchester and the Cambridge Biomedical Campus, while also emphasizing the importance of open spaces and community centers. Therefore, the summary does not fully align with the source document and contains hallucinations.
+
+**Score**: FAIL → Iterate to create another summary using the hallucination agent response
+
+**Final**:
+
+The response opposes the development of the area east of Milton Road, arguing that the wastewater treatment plant is fit for purpose and should not be relocated due to high costs. It supports the development of the Cambridge Biomedical Campus with a focus on vast open spaces and community centers. Cambourne should remain isolated and self-sufficient, with an emphasis on cycle routes. The response opposes development in the southern rural cluster of villages, supports limited development in Grantchester, and suggests that road bypasses are needed. It identifies Barton and Newnham as potential sites for development. The overall vision for Greater Cambridge in 2041 emphasizes preservation and limited development.
+
+

reports/DOCS/_extensions/nrennie/PrettyPDF/PrettyPDF.lua

@@ -0,0 +1,10 @@
+function Div(el)
+  if el.classes:includes('article') then
+    local blocks = pandoc.List({
+      pandoc.RawBlock('typst', '#article[')
+    })
+    blocks:extend(el.content)
+    blocks:insert(pandoc.RawBlock('typst', ']\n'))
+    return blocks
+  end
+end

reports/DOCS/_extensions/nrennie/PrettyPDF/PrettyPDF.tex

@@ -0,0 +1,77 @@
+% load packages
+\usepackage{geometry}
+\usepackage{xcolor}
+\usepackage{eso-pic}
+\usepackage{fancyhdr}
+\usepackage{sectsty}
+\usepackage{fontspec}
+\usepackage{titlesec}
+
+%% Set page size with a wider right margin
+\geometry{a4paper, total={170mm,257mm}, left=20mm, top=20mm, bottom=20mm, right=50mm}
+
+%% Let's define some colours
+\definecolor{light}{HTML}{E6E6FA}
+\definecolor{highlight}{HTML}{800080}
+\definecolor{dark}{HTML}{330033}
+
+%% Let's add the border on the right hand side 
+\AddToShipoutPicture{% 
+    \AtPageLowerLeft{% 
+        \put(\LenToUnit{\dimexpr\paperwidth-3cm},0){% 
+            \color{light}\rule{3cm}{\LenToUnit\paperheight}%
+          }%
+     }%
+     % logo
+    \AtPageLowerLeft{% start the bar at the bottom right of the page
+        \put(\LenToUnit{\dimexpr\paperwidth-2.75cm},27.2cm){% move it to the top right
+            \color{light}\includegraphics[width=2.5cm]{_extensions/nrennie/PrettyPDF/logo.png}
+          }%
+     }%
+}
+
+%% Style the page number
+\fancypagestyle{mystyle}{
+  \fancyhf{}
+  \renewcommand\headrulewidth{0pt}
+  \fancyfoot[R]{\thepage}
+  \fancyfootoffset{3.5cm}
+}
+\setlength{\footskip}{20pt}
+
+%% style the chapter/section fonts
+\chapterfont{\color{dark}\fontsize{20}{16.8}\selectfont}
+\sectionfont{\color{dark}\fontsize{20}{16.8}\selectfont}
+\subsectionfont{\color{dark}\fontsize{14}{16.8}\selectfont}
+\titleformat{\subsection}
+  {\sffamily\Large\bfseries}{\thesection}{1em}{}[{\titlerule[0.8pt]}]
+  
+% left align title
+\makeatletter
+\renewcommand{\maketitle}{\bgroup\setlength{\parindent}{0pt}
+\begin{flushleft}
+  {\sffamily\huge\textbf{\MakeUppercase{\@title}}} \vspace{0.3cm} \newline
+  {\Large {\@subtitle}} \newline
+  \@author
+\end{flushleft}\egroup
+}
+\makeatother
+
+%% Use some custom fonts
+\setsansfont{Ubuntu}[
+    Path=_extensions/nrennie/PrettyPDF/Ubuntu/,
+    Scale=0.9,
+    Extension = .ttf,
+    UprightFont=*-Regular,
+    BoldFont=*-Bold,
+    ItalicFont=*-Italic,
+    ]
+
+\setmainfont{Ubuntu}[
+    Path=_extensions/nrennie/PrettyPDF/Ubuntu/,
+    Scale=0.9,
+    Extension = .ttf,
+    UprightFont=*-Regular,
+    BoldFont=*-Bold,
+    ItalicFont=*-Italic,
+    ]

reports/DOCS/_extensions/nrennie/PrettyPDF/Ubuntu/UFL.txt

@@ -0,0 +1,96 @@
+-------------------------------
+UBUNTU FONT LICENCE Version 1.0
+-------------------------------
+
+PREAMBLE
+This licence allows the licensed fonts to be used, studied, modified and
+redistributed freely. The fonts, including any derivative works, can be
+bundled, embedded, and redistributed provided the terms of this licence
+are met. The fonts and derivatives, however, cannot be released under
+any other licence. The requirement for fonts to remain under this
+licence does not require any document created using the fonts or their
+derivatives to be published under this licence, as long as the primary
+purpose of the document is not to be a vehicle for the distribution of
+the fonts.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this licence and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Original Version" refers to the collection of Font Software components
+as received under this licence.
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to
+a new environment.
+
+"Copyright Holder(s)" refers to all individuals and companies who have a
+copyright ownership of the Font Software.
+
+"Substantially Changed" refers to Modified Versions which can be easily
+identified as dissimilar to the Font Software by users of the Font
+Software comparing the Original Version with the Modified Version.
+
+To "Propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification and with or without charging
+a redistribution fee), making available to the public, and in some
+countries other activities as well.
+
+PERMISSION & CONDITIONS
+This licence does not grant any rights under trademark law and all such
+rights are reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of the Font Software, to propagate the Font Software, subject to
+the below conditions:
+
+1) Each copy of the Font Software must contain the above copyright
+notice and this licence. These can be included either as stand-alone
+text files, human-readable headers or in the appropriate machine-
+readable metadata fields within text or binary files as long as those
+fields can be easily viewed by the user.
+
+2) The font name complies with the following:
+(a) The Original Version must retain its name, unmodified.
+(b) Modified Versions which are Substantially Changed must be renamed to
+avoid use of the name of the Original Version or similar names entirely.
+(c) Modified Versions which are not Substantially Changed must be
+renamed to both (i) retain the name of the Original Version and (ii) add
+additional naming elements to distinguish the Modified Version from the
+Original Version. The name of such Modified Versions must be the name of
+the Original Version, with "derivative X" where X represents the name of
+the new work, appended to that name.
+
+3) The name(s) of the Copyright Holder(s) and any contributor to the
+Font Software shall not be used to promote, endorse or advertise any
+Modified Version, except (i) as required by this licence, (ii) to
+acknowledge the contribution(s) of the Copyright Holder(s) or (iii) with
+their explicit written permission.
+
+4) The Font Software, modified or unmodified, in part or in whole, must
+be distributed entirely under this licence, and must not be distributed
+under any other licence. The requirement for fonts to remain under this
+licence does not affect any document created using the Font Software,
+except any version of the Font Software extracted from a document
+created using the Font Software may only be distributed under this
+licence.
+
+TERMINATION
+This licence becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
+COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER
+DEALINGS IN THE FONT SOFTWARE.

reports/DOCS/_extensions/nrennie/PrettyPDF/_extension.yml

@@ -0,0 +1,30 @@
+title: PrettyPDF
+author: Nicola Rennie
+version: 0.0.3
+contributes:
+  project:
+    project:
+      type: book
+  formats: 
+    pdf:
+      include-in-header: 
+       - "PrettyPDF.tex"
+      include-before-body:
+       - "pagestyle.tex"
+      toc: false
+      code-block-bg: light
+      linkcolor: highlight
+      urlcolor: highlight
+    typst:
+      papersize: a4
+      margin:
+        x: 2cm
+        y: 2cm
+      font-paths: Ubuntu
+      typst-logo: 
+        path: "logo.png"
+      template-partials:
+        - typst-template.typ
+        - typst-show.typ
+      filters:
+        - PrettyPDF.lua

reports/DOCS/_extensions/nrennie/PrettyPDF/pagestyle.tex

@@ -0,0 +1 @@
+\pagestyle{mystyle}

reports/DOCS/_extensions/nrennie/PrettyPDF/typst-show.typ

@@ -0,0 +1,15 @@
+#show: PrettyPDF.with(
+$if(title)$
+  title: "$title$",
+$endif$
+$if(subtitle)$
+  title: "$subtitle$",
+$endif$
+$if(typst-logo)$
+  typst-logo: (
+    path: "$typst-logo.path$",
+    caption: [$typst-logo.caption$]
+  ), 
+$endif$
+)
+

reports/DOCS/_extensions/nrennie/PrettyPDF/typst-template.typ

@@ -0,0 +1,90 @@
+
+#let PrettyPDF(
+  // The document title.
+  title: "PrettyPDF",
+
+  // Logo in top right corner.
+  typst-logo: none,
+
+  // The document content.
+  body
+) = {
+
+  // Set document metadata.
+  set document(title: title)
+  
+  // Configure pages.
+  set page(
+    margin: (left: 2cm, right: 1.5cm, top: 2cm, bottom: 2cm),
+    numbering: "1",
+    number-align: right,
+    background: place(right + top, rect(
+      fill: rgb("#E6E6FA"),
+      height: 100%,
+      width: 3cm,
+    ))
+  )
+  
+  // Set the body font.
+  set text(10pt, font: "Ubuntu")
+
+  // Configure headings.
+  show heading.where(level: 1): set block(below: 0.8em)
+  show heading.where(level: 1): underline
+  show heading.where(level: 2): set block(above: 0.5cm, below: 0.5cm)
+
+  // Links should be purple.
+  show link: set text(rgb("#800080"))
+
+  // Configure light purple border.
+  show figure: it => block({
+    move(dx: -3%, dy: 1.5%, rect(
+      fill: rgb("FF7D79"),
+      inset: 0pt,
+      move(dx: 3%, dy: -1.5%, it.body)
+    ))
+  })
+
+  // Purple border column
+  grid(
+    columns: (1fr, 0.75cm),
+    column-gutter: 2.5cm,
+
+    // Title.
+    text(font: "Ubuntu", 20pt, weight: 800, upper(title)),
+
+    // The logo in the sidebar.
+    locate(loc => {
+      set align(right)
+
+      // Logo.
+      style(styles => {
+        if typst-logo == none {
+          return
+        }
+  
+        let img = image(typst-logo.path, width: 1.5cm)
+        let img-size = measure(img, styles)
+        
+        grid(
+          columns: (img-size.width, 1cm),
+          column-gutter: 16pt,
+          rows: img-size.height,
+          img,
+        )
+      })
+      
+    }),
+    
+    // The main body text.
+    {
+      set par(justify: true)
+      body
+      v(1fr)
+    },
+  
+
+  )
+}
+
+