arxiv:2604.05404

Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning

Published on Apr 7

· Submitted by

SII-sqs on Apr 8

University of Science and Technology of China

Upvote

Authors:

Qisheng Su ,

Zhen Fang ,

Abstract

Researchers introduce PTE (Prefill Token Equivalents), a hardware-aware metric for measuring efficiency in Tool-Integrated Reasoning scenarios, which better correlates with actual inference latency than traditional token counts by accounting for KV-Cache inefficiencies and long tool responses.

AI-generated summary

In real-world Tool-Integrated Reasoning (TIR) scenarios, where LLMs interleave reasoning with external tool calls, a major source of inefficiency is that the toolcalls create pauses between LLM requests and cause KV-Cache eviction, forcing recomputation. Also, the long, unfiltered response returned by external tools inflates the KV-Cache, so each decode step spends more time loading the growing cache and thus becomes steadily slower as context length increases. However, existing efficiency metrics like token counts and toolcall counts fail to capture the real model inference latency. To address this, we introduce PTE (Prefill Token Equivalents), a hardware-aware TIR-efficiency metric that unifies internal reasoning and external tool-use costs while explicitly accounting for non-reusable KV-Cache and long-tool-response scenarios. Validation in a high-concurrency industrial setting indicates that PTE aligns significantly better with wall-clock latency than standard token counts, while maintaining consistent efficiency rankings across diverse hardware profiles. We conduct extensive experiments across five TIR benchmarks, quantify their PTE costs, and identify four inefficiency patterns that appear in TIR. We also discover that trajectories with higher PTE costs tend to have lower reasoning correctness, indicating that simply using more tools does not improve the quality of the answer.

View arXiv page View PDF GitHub 24 Add to collection

Community

groundhogLLM

Paper author Paper submitter about 14 hours ago

In real-world Tool-Integrated Reasoning (TIR) scenarios, a major source of inefficiency is that the toolcalls create pauses between LLM requests and cause KV-cache eviction. Also, the long, unfiltered response returned by external tools inflates the KV-cache, so each decode step spends more time loading the growing cache and thus becomes steadily slower as context length increases. However, existing efficiency metrics like token counts and toolcall counts fail to capture this real computational cost. To address this, we introduce PTE (Prefill Token Equivalents), a hardware-aware TIR-efficiency metric that unifies internal reasoning and external tool-use costs while explicitly accounting for non-reusable KV-Cache and long-tool-response scenarios, thus better reflects real-world scenarios. We conduct extensive experiments across five TIR benchmarks, quantify their PTE costs, and identify four inefficiency patterns that appear in TIR. In a simulated high-concurrency industrial setting, PTE explains wall-clock latency significantly better than token-count metric. We also discover that trajectories with higher PTE costs tend to have lower reasoning correctness, indicating that simply using more tools does not improve the quality of the answer. PTE offers a new perspective on the efficiency of Tool-Integrated Reasoning. The code is available.

avahal

about 4 hours ago

Interesting breakdown of this paper on arXivLens: https://arxivlens.com/PaperView/Details/beyond-accuracy-unveiling-inefficiency-patterns-in-tool-integrated-reasoning-3556-2c57f15c
Covers the executive summary, detailed methodology, and practical applications.

mishig

43 minutes ago

Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning

Tool-Integrated Reasoning (TIR) lets language models call external tools such as code interpreters, but each tool call triggers KV-Cache eviction and inflation that existing efficiency metrics (token count, number of tool calls) fail to capture. This paper proposes PTE (Prefill Token Equivalents), a hardware-aware metric that unifies the cost of internal reasoning and external tool use, and identifies four recurring inefficiency patterns in TIR systems.

Key Idea

Every external tool call forces the model to evict cached key-value states and re-prefill context, creating latency spikes invisible to simple token-counting metrics. PTE converts both internal generation and external tool-call overhead into a single comparable unit — the equivalent number of prefill tokens — giving a faithful picture of real wall-clock cost on actual hardware.

Method / Approach

PTE is computed by profiling the actual prefill and decode costs on target hardware, then mapping each reasoning step and tool invocation to its prefill-token equivalent. Using PTE, the authors analyze a range of TIR-enabled models and identify four distinct inefficiency patterns: redundant tool calls, excessive context re-prefilling, unnecessary chain-of-thought before tool use, and repeated failed invocations.

Results

Across multiple benchmarks, higher PTE cost correlates with lower correctness — models that spend more compute on tool interactions tend to produce worse answers. This suggests that current TIR systems waste substantial resources on unproductive tool calls, and that optimizing for PTE could simultaneously improve both efficiency and accuracy.