LLoCO: Learning Long Contexts Offline
Paper
•
2404.07979
•
Published
•
20
Long Context
-
-
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Paper • 2402.13753 • Published • 114 -
LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
Paper • 2402.11550 • Published • 16 -
LongAlign: A Recipe for Long Context Alignment of Large Language Models
Paper • 2401.18058 • Published • 20 -
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Paper • 2404.07143 • Published • 105 -
Long-Context Language Modeling with Parallel Context Encoding
Paper • 2402.16617 • Published • 1 -
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Paper • 2406.10149 • Published • 49 -
RULER: What's the Real Context Size of Your Long-Context Language Models?
Paper • 2404.06654 • Published • 34 -
Extending Llama-3's Context Ten-Fold Overnight
Paper • 2404.19553 • Published • 33 -
Make Your LLM Fully Utilize the Context
Paper • 2404.16811 • Published • 52 -
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
Paper • 2404.08801 • Published • 64 -
LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
Paper • 2406.00605 • Published • 2 -
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
Paper • 2402.02244 • Published • 1 -
Resonance RoPE: Improving Context Length Generalization of Large Language Models
Paper • 2403.00071 • Published • 22 -
Extending LLMs' Context Window with 100 Samples
Paper • 2401.07004 • Published • 15 -
Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
Paper • 2406.11230 • Published • 33 -
Long Code Arena: a Set of Benchmarks for Long-Context Code Models
Paper • 2406.11612 • Published • 24 -
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Paper • 2406.16008 • Published • 6 -
LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs
Paper • 2406.15319 • Published • 62 -
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
Paper • 2406.16747 • Published • 18 -
Can Few-shot Work in Long-Context? Recycling the Context to Generate Demonstrations
Paper • 2406.13632 • Published • 5 -
LongIns: A Challenging Long-context Instruction-based Exam for LLMs
Paper • 2406.17588 • Published • 23 -
Training-Free Long-Context Scaling of Large Language Models
Paper • 2402.17463 • Published • 19 -
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
Paper • 2406.17419 • Published • 17 -
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
Paper • 2405.17915 • Published • 2 -
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Paper • 2407.01370 • Published • 86 -
Human-like Episodic Memory for Infinite Context LLMs
Paper • 2407.09450 • Published • 60 -
NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context Window?
Paper • 2407.11963 • Published • 43 -
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
Paper • 2407.14057 • Published • 45 -
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
Paper • 2407.14482 • Published • 26 -
Writing in the Margins: Better Inference Pattern for Long Context Retrieval
Paper • 2408.14906 • Published • 139 -
LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA
Paper • 2409.02897 • Published • 45 -
LongRecipe: Recipe for Efficient Long Context Generalization in Large Languge Models
Paper • 2409.00509 • Published • 38 -
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Paper • 2409.16191 • Published • 42 -
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval
Paper • 2409.10516 • Published • 41 -
Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models
Paper • 2409.04774 • Published -
L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?
Paper • 2410.02115 • Published • 10 -
Minimum Tuning to Unlock Long Output from LLMs with High Quality Data as the Key
Paper • 2410.10210 • Published • 5 -
LongReward: Improving Long-context Large Language Models with AI Feedback
Paper • 2410.21252 • Published • 17 -
Why Does the Effective Context Length of LLMs Fall Short?
Paper • 2410.18745 • Published • 17 -
Language Models can Self-Lengthen to Generate Long Texts
Paper • 2410.23933 • Published • 17 -
Large Language Models Can Self-Improve in Long-context Reasoning
Paper • 2411.08147 • Published • 63 -
Star Attention: Efficient LLM Inference over Long Sequences
Paper • 2411.17116 • Published • 48 -
LongKey: Keyphrase Extraction for Long Documents
Paper • 2411.17863 • Published • 11 -
Reverse Thinking Makes LLMs Stronger Reasoners
Paper • 2411.19865 • Published • 20 -
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?
Paper • 2411.05000 • Published • 21 -
How to Train Long-Context Language Models (Effectively)
Paper • 2410.02660 • Published • 2 -
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
Paper • 2411.13476 • Published • 15 -
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
Paper • 2408.07055 • Published • 66 -
Differential Transformer
Paper • 2410.05258 • Published • 169