Drowning in Documents: Consequences of Scaling Reranker Inference

Rerankers (cross-encoders) and retrievers (embeddings) are often derived from the same architecture, yet rerankers are assumed to be more accurate given that they jointly encode the query and document rather than process them independently. In this work, we find two surprising results with respect to this intuition: 1. reranking helps at first, but eventually reranking too many documents leads to a decrease in quality, and 2. in a fair match up between rerankers and retrievers where we rerank the full dataset, rerankers are less accurate than retrievers. In our paper we detail extensive experiments across both academic and enterprise datasets, and include results that suggest listwise reranking with LLMs are more robust than cross-encoders when scaling inference via reranking.