Tool Calls Double Your Latency. Did You Load-Test Them?

The math nobody does before shipping an agent

Tool calling, function calling, feels free in a demo. You ask the assistant a question, it quietly calls a weather API or a database, and answers. One question, one answer. It looks like a single interaction.

Under the hood it is not. A tool-calling turn is a loop. The model reads your prompt and decides to call a function (LLM call #1). Your application executes that function (a network call to an API or a query to a database). You append the result to the conversation and call the model again so it can read the result and produce an answer (LLM call #2). That is two LLM invocations and one tool execution for the simplest possible case, and each LLM invocation pays its own time-to-first-token over a context that grew with every step. As practitioner writing on how poor tool calling increases LLM cost and latency puts it bluntly, each tool invocation requires a full model inference cycle, so 20 sequential tool calls means 20 inference round trips.

# End-to-end latency of a tool-calling turn (serial)
total = (
    llm_ttft_1 + llm_decide   # model reads prompt, emits the tool call
  + tool_exec                 # your API/DB does the work
  + llm_ttft_2 + llm_answer   # model reads the result, writes the reply
)

# A plain completion is just:  llm_ttft + llm_answer
# So even ONE tool roughly DOUBLES the LLM work and adds the tool's own latency.

Now chain it. Real agents rarely call one tool. They look something up, then call another tool based on what they found, then maybe a third. A 2026 analysis of parallel tool-calling optimization makes the cost concrete: a four-tool sequence where each call takes 300ms is 1.2 seconds of dead time when run serially, collapsing to 300ms in parallel, and where tools do not depend on each other, serializing them is "pure latency with no benefit." But where step N+1 genuinely depends on step N's result, the calls cannot be parallelized and the latencies add. A four-tool dependent loop is, latency-wise, five LLM calls and four tool executions stacked end to end.

Why each round trip is more expensive than the last

Tool calling does not just multiply requests; it inflates each one. Every round trip carries forward the full conversation: the original prompt, the tool schemas, the model's tool-call message, and the tool's (often verbose) result. That context grows monotonically, and because time-to-first-token is prefill-bound, it scales with input length, each successive LLM call in the loop starts slower than the one before. NVIDIA's inference fundamentals are explicit that prefill cost rises with input tokens; an agent loop is a machine for making inputs longer on every iteration.

The tool schemas themselves are a hidden tax. If you expose twenty tools, the JSON schema for all twenty is in the prompt on every call in the loop, even the ones where the model only needs one. The schemas count against the context window on each hop, so keeping the number of tools small and descriptions tight is not just an accuracy practice, it is a latency lever, because fewer, leaner schemas mean less to prefill on every round trip.

The tool execution is the part that betrays you under load

Here is the failure that actually ships. The LLM provider's API is built to scale and absorb concurrency. Your tools usually are not. A function call hits your internal microservice, your Postgres, a third-party API with a rate limit, a search index. Those are precisely the systems whose tails fatten under load, and the agent loop multiplies your exposure to them, because one user turn might fire three or four tool calls, and a hundred concurrent users fire three or four hundred.

So you can have an LLM that streams beautifully under load and an agent that still feels broken, because the tool execution stage is queueing behind a database connection pool that was never sized for agent-driven fan-out. The classic tail-amplification result from Dean and Barroso's "The Tail at Scale" applies in full: a turn is slow if any tool call in its chain is slow, and chaining several tools per turn multiplies the chance of catching a tail.

Measuring the loop, not the pieces

Instrument every hop in the tool-calling turn so you can see which link dominates, and crucially, capture how the breakdown shifts as concurrency rises:

# Per-turn agent latency breakdown
import time

def run_agent_turn(client, messages, tools):
    spans, t = [], time.perf_counter()
    def lap(label):
        nonlocal t
        now = time.perf_counter(); spans.append((label, (now - t) * 1000)); t = now

    while True:
        resp = client.chat(messages=messages, tools=tools)   # LLM hop
        lap('llm')
        calls = resp.tool_calls
        if not calls:
            lap('final'); break
        for c in calls:
            result = dispatch(c)                              # tool hop (YOUR system)
            messages.append(tool_message(c, result))
        lap('tools')
        messages.append(resp.message)

    total = sum(ms for _, ms in spans)
    return total, spans   # e.g. [('llm', 310), ('tools', 840), ('llm', 520), ('final', 410)]

Then run that turn under concurrency and report the distribution of the total and of each stage, so you know whether the LLM hops or the tool hops blow up first:

import numpy as np
from collections import defaultdict

def agent_load_report(turns, budget_ms):
    totals = [sum(ms for _, ms in s) for s in turns]
    p50, p95, p99 = np.percentile(totals, [50,95,99])
    print(f"agent turn  p50 {p50:.0f}  p95 {p95:.0f}  p99 {p99:.0f} ms")

    by_stage = defaultdict(list)
    for s in turns:
        for label, ms in s: by_stage[label].append(ms)
    for label, xs in by_stage.items():
        print(f"  {label:6} p95 {np.percentile(xs, 95):.0f} ms  (which hop dominates?)")

    over = np.mean(np.array(totals) > budget_ms) * 100
    print(f"{over:.1f}% of turns exceed {budget_ms}ms budget")
    print("PASS" if p95 <= budget_ms else "FAIL, tool loop blows the budget")

How to spend less in the loop

• Parallelize independent tool calls. If the model requests three tools that do not depend on each other in one step, execute them concurrently instead of serially. Every major provider, OpenAI, Anthropic, Google, now ships native parallel function calling, and the parallel-tool-calling research above reports up to 3.7x latency speedup and 6.7x cost reduction versus ReAct-style sequential execution. Use it.
• Cut the round trips. Fewer, more capable tools beat many narrow ones. A single "search_and_summarize" call can replace a search-then-read-then-summarize chain of three.
• Cache the static prefix. Tool schemas and the system prompt are identical on every hop; prompt caching can skip re-prefilling them, directly cutting each round trip's TTFT.
• Bound the loop. Cap the number of tool-calling iterations. An agent that can loop indefinitely will, under the wrong input, produce a latency outlier that is effectively a timeout.
• Stream the final answer. The last LLM hop can stream to the user even while you log the rest, get the first token of the answer out fast.

Gate on the agent turn

Make the end-to-end tool-calling turn a release gate, measured under concurrency, with a separate assertion on the tool stage so a slow downstream system cannot hide behind a fast LLM:

assert agent_p95_ms      < 4000, f"agent turn p95 over budget: {agent_p95_ms}ms"
assert agent_p99_ms      < 8000, f"agent turn p99 tail too long: {agent_p99_ms}ms"
assert tool_stage_p95_ms < 1500, f"tool execution is the bottleneck: {tool_stage_p95_ms}ms"
assert max_tool_loops    <= 6,   "agent looped too many times, latency outlier risk"

Now adding a new tool that bloats every prefill, swapping in a slower downstream API, removing the parallel-execution path, or a concurrency level your database pool cannot sustain all turn into a red build, instead of an agent that demos fast and crawls in production.

The bottom line

Tool calling is not a free feature bolted onto a completion, it is a serial loop of LLM calls and external calls, each LLM hop re-paying prefill over a growing context and each tool hop inheriting the tail of a system you also have to scale. One tool roughly doubles the LLM work; a four-tool serial chain is over a second of dead time, and none of it parallelizes when the steps depend on each other. The latency that ships is the loop's, and almost nobody load-tests the loop. Instrument every hop, run the whole turn under concurrency, watch which stage blows up first, and gate on the end-to-end budget. Then your agent stays as fast in production as it was in the demo.