Note Top reasoning trick on HummanEval: MCTS + LLM + Feedback + Reflection @UIUC

Note Our Re-Implementation code: https://github.com/fangyuan-ksgk/CoT-Reasoning-without-Prompting Insight: Decoding time reasoning is cheap, effective, and can bring out the 'inherent' reasoning capacity from pre-trained LLM. Drawback: Indentification of the set of answer, and its location reamains the million dollar question.

Note In-Context-Learning based preference alignment, performance on-par with Supervised Fine-Tuning (SFT). Can be used to generated optimal preference pairs, or augment the preference dataset.

Note Self-Discover solves any task in three steps: Pickging a reasoning structure, designing a stepwise reasoning plan, then implement the thinking process to get the answer. Significant performance improvement is observed.

Note Meta's work on iterative self-improvement of LLM.

Note A simplification of Meta's self-rewarding LLM, relying on LLM's innate capacity of understanding the preference shown in the original labeled dataset, and use it to gives thumb up & down, which are then feed back to the model weight through DPO.

Note Matryoshka embedding innovatively blends PCA's flexibility with the precision of learned embeddings. It achieves this with a simple change on the loss function. Instead of only aiming for a high-quality D-dimensional compression, the Matryoshka approach requires that truncated portions of this embedding - like the first 2, 4, ..., up to D/2 dimensions - also be effective in isolation. This dual focus ensures that even smaller segments of the embedding retain useful information. The intuition her

Note In-context learning by day & fine-tuning by night. This will rock the back-prop learning tradition, and achieve human-level efficient learning capacity.

Note Essentially extrapolate the rotary position embedding adaptively, with some evolution algorithm to my vague understanding, not thoroughly read yet. Although seems to be MC's response to Google's RingAttention

Note May the Source be with you :>

Note Updated Version of Fine-Tuning guide from Google.

Note DeepMind's prompt based reasoning approach: Monte Carlo Tree, Breadth first search / Depth first search, deliberate atomic thought process controlled with proposal prompt. Essentially these guys are doing chess here.

Note "...the presented experiments use token sequences that are significantly longer than the sequences used to train LLMs such as Llama 2..." More computing resources than what they put into llama2 is now poured into training on a --maze game..... This simply can't be the way towards reasoning capacity guys.

Note For obvious reasons, we have to read this one.

Note This is the foundation of modern accelarated attention mechanism. Softmax requires a normalization constant to be computed in order to get the normalized attention weights for each value. A simple & obvious observation says that global normalization constant can be ensembled from two sub-chunks' computation result, therefore parallel computation among chunks together with a final ensemble steps allow us to accelerate attention computation, which leads to efficient attention mechanism.

Note Similar idea as the online-softmax paper, parallelism is achieved by breaking sequence length (number of queries) into chunks, the parallelism of softmax computation means such chunking operation would be equivalent to the full attention mechanism, whilst allowsing for speed-up through multiprocessing hardrwares like GPU/TPU.

Note Re-write of the efficient memory attention mechanism, every query, key, value is separated into blocks (along their sequence length dimension). Merge is conducted after parallel computation of normalization constant and attention weights & values on each block. An observation is made that the feedforward propagation, and the residual operation is also parallelizable and can be done within each query block. key & value block attention -> query block merge attention, feedforward & residual is done

Note Sacrifice latency to gain memory efficiency compared to BPT, by using Ring-communication to restrict communication to 1-copy per host, instead of passing all parallel computation result into one device (highly memory requiring). This lossens the memory requirement to achieve higher context length, therefore scaling up the context-length of transformer to 10M tokens.

Note UC Berkeley rocks

Note This is the preliminary technique behind LongRoPE. Rotary position embedding addresses the crave for 'relative position embedding' through independent embedding with block-wise diagonal rotation matrix. Microsoft's approach to scaling their attention context, is through extrapolation / interpolation based off this rotatry embedding vectors.

Note Rotation contains much more entropy than scaling, and it is much more friendly to combination, and less-prune to explosion / vanishing values. Rotating the neural network parameters just seems much more important than scaling them. Eventually people might just converge towards binary paramter, where you do not need to scale anything.

Note Confirmed to be real by open-source people -- learn this as the new bible if you wish to pre-train your own model. Linear layer from any ViT / Transformer can be replaced by a BitNet with teneary parameters and achieve similar performance via quantization-aware training. Re-Implementation with MoE-BitNet too: https://github.com/fangyuan-ksgk/1bitNet So there is free lunch? Or just that float16 is too redundant with so many layers?

Note UC Berkeley rocks

Note This work is similar to ICDPO, trying to utilize the knowledge inside LLM through Contrastive Decoding (CD). Linear extrapolation is adopted to align the logits according to the direction of alignment, calculated by the difference in logits between a preference principle prompted LLM and a plain LLM.

Note Essentially LLM can be viewed as a perplexity score refiner, given any shit & non-sence setence, it can re-predict a better sentence with simple casual masking, which has better perplexity score, the trick is such casual masking prediction is naturally possible to carried out in parallel. Then you have the problem of requiring too much iterations, or dropping accuracy because you are conditioning on shit etc... Therefore some fine-tuning tricks are applied to force out more performance.

Note OG work from openAI, decouple weight manitude & direction for learning. Everything boils down to one equation.

Note The Heyna operator utilizes long convolution to achieve better performance than the attention operator at a fraction of the cost, particularly with longer contexts. Interestingly, the convolution filter is controlled by the position of the token, which also allows for a structured window to control the attention mechanism. This enables behaviors such as decay-through-time, which more closely resembles human attention.

Note something like 'we use python'

Note parallel decoding for parallel thoughts, sequential decoding for serial deduction process

Note CoT but with step-wise RAG (bit crazy but multi-processing allows parallel RAG implementation, so why not?)

Note Trade more computation for less memory. Extra computation involves dequantizing the quantized weight matrices, extra gradient propagation into low-rank adapter matrices. Less memory means no need to store the pre-trained float16 weight matrix, and no need to store gradient for the pre-trained weight matrix. There is no free lunch. BTW, fine-tuning a X GB model requires 4X GB memory to store weights and gradients. LoRA achieves ~XGB by using adaptors, and QLoRA ahieves ~X/N GB with quantization.

Note work from KAUST optimizes Graph-based Agent cooperation

Note How to hack OpenAI

Note Pre-train on 24GB GPU !! We know LoRA part, but what is Ga?

Note Predecessor work for era of 1-bit LLM paper, here they actually scale everything down to 1-bit, even the activation function gets quantized. One could argue that the quantization of the activation function destructs the performance of these quantization-aware trained model.

Note In-context RL:

Note This is the foundation of modern time fine-tuning & quantization, and the experimental discovery here closely relate to theory of mind. Essentially with more paramters, model will be more reluctant to learn with high intrinsic dimensional, as then a lower dimensional representation suffices with the shear amount of parameters. As a result, model is always trying to minimize 'surprise' while reducing 'energy' -- in this case, the intrinsic dimension.

Note Branch: copy base LLM multiple times. Train: Shard training data into multiple domain-specific subset, and train each LLM in parallel (zero communication required). Mix: Merge all domain specific LLMs into a MoE model (token-level routing + multiple FFN layers), then MoE-finetune this model to adapt to the full training dataset. Interesting discussion on dying expert (routed weights collapse to zero) effect and the 'load balancing' tricks as a simple solution by adding balancing loss.

Note Goal-Oriented Model Merging with Evolutionary Search. PS (slerp / dare -- succeeds) + DFS (passthrough with scaler adaptor -- fails!) Our re-implementation here: https://github.com/fangyuan-ksgk/Evolutionary-Model-Merge

Note 'Pathway' vision at Google & Imaginary work from Deepmind: ML Model as decomposable pathways, each path is capcable of process input and provide output, with this model structure, distributed training across different clusters require low synchronization and communication. Inference with such model also require only loading the few offline-routed paths which matter to your task at hand, alleviating the storage requirement of the computation.

Note Simple yet powerful observation: noise get included into RAG pipeline, so fine-tune your LLM against such noise through data augmentation.

Note Successful Data Flow merging, two Mistral 7B is adopted to concatenate with each other with overlap, the overlapped layers gets merged, and continual pre-training is done while observing "fast performance recovery". It seems the nice LLM has much more compressed & generalizble information stored in each layer than we think. Contiual pretraining from a stack of these layers might be a good idea for efficient scaling of model size.

Note Split, Revise to be self contained ensure the atomic thought is maintained, and the argument can be broken down into coherant small arguments. Based on the smaller argument, one can do fact check, do reflection, do MCTS, do all sorts of stuff with them. This little trick opens a door to immense posibility. And specifically useful with their open-sourced code !

Note An extension to the Branch-Train-Mix approach from Meta. Multiple copies are made from the base LLM, a unsupervised clustering-based domain discovery (main difference) is done by traversing the training data corpus, and then offline task-level routing ensures each LLM is trained into a domain experts. Mixing weighted logit prediction from routed top-K experts is carried out during inference.

Note Google's work on Multilingual Machine Translation task in 2021. Token-MoE suffer from having to load all the experts in storage while dynamic token routing only activate a few of them -- this creates redundancy. By assigning each input language & output language pair as a separate task, task-level routing (Task-MoE) is shown to outperform Token-MoE on multilingual machine translation task, avoiding interference by mixed training.

Note It seems Meta is not playing RL in the right way, check Quiet-StaR, a significant improvement is possible. Again one potential reson is that the thought generation process in here is not PARALLEL. As Matt puts it, thinking process is fundamentally Parallel, while the saying process isn't. So the RL should be done with the thought, and not the greedy-1 decoding path? Need to dig it

Note LLM OS == fine-tuning on specific sequence of actions within a constrained environment == few billiion parameter agent ?

Note 'Skip' transformer blocks yields on par performance than sequential propagation without 'skipping'. This is essentially a 'Hard' residual, as opposed to the 'Soft' residual in ResNet.

Note Given a pre-trained LLM, sequence input is represented in latent space across transformer blocks. These representation can be intervened to achieve 'control' of model output, which higher efficiency than modification of model weights. Representation fine-tuning uses learnable representation intervention for fine-tuning, achieving 10x-50x more efficiency boost compared to LoRA & DoRA. Compositional learning is done by doing mutually exclusive intervention on separate position & layers.

Note Iterative DPO. Difference with self-rewarding LLM is that it uses a strong LLM to RANK the generated responses, thereby having a much better efficiency in 'aligning' the small / weak model towards the strong model. Moreover, the nature of evaluation being simpler than generation means one could even surpass GPT-4 performance with such iteration schemes.

Note Position paper on data augmentation approaches adopted in LLM training. Real data is expensive to get and will run out very quickly, most of the public benchmark dataset is already inside training corpus for many LLMs. One interesting point: replacing natural language label with random labels improve model performance. Classic improving difficulty leads to generalization trick.

Note Residual Connection has been proven to be the right way to compose neural network layers. This work from Peking university proves the same trick can be used to compose LLMs. (Not surprising at all). The image reminds me of the quote from Issac Newton: "if I have seen further, it is by standing on the shoulders of giants"

Note This work touches on the fundamental issue of ML model training: it always pick the short-cut. To predict natural language label, the short-cut is to check semantic similarity. But what if we replace the natural language label with random labels? Then the model is forced to 'learns' what it shoudl do, and then do it.

Note Not the right way to go. Model would reinforce itself.

Note Equivalence between optimal compression and perplexity score. LLM encodes rare sequence with more bits and common sequence with less bits, the less bits it required, the more compressed the encoding is. Pre-Training loss is precisely modeling such compression efficiency. This paper identifies a linear correpondance of comression and Intelligence.

Note Increase the amount of few-shot prompts to >500k. Interesting discovery is that "Unsupervised ICL" which presents problem and not problem-solution pairs achieves on-par performance with question-answer prompting. We need to present context to LLM such that its domain-specific knowledge could be revealed.

Note Observation is made that LLM errors mostly because it fails to understand the question. Rephrasing the question by LLM itself provides explicit requirement for LLM to address this error, combination of question rephrasing, information extraction and then step-by-step reasoning leads to higher accuracy. More compute, more accuracy.

Note Get error message and re-program is generally what happens when we code. However, this is not exactly what happens, error message help us locate the exact place where the error occurs, which helps breaking down the code issue, and allow one to trace back the issue. This paper sort of does that.