Multimodal Load Testing: Big Payloads Break Small Assumptions

The demo used a thumbnail. Production gets 12-megapixel photos.

Here is how the assumption breaks. The team builds an image-understanding feature, tests it with the handful of small sample images in the repo, sees fast responses and a tidy token bill, and ships. Then real users arrive, and real users upload whatever their phone produced: a 12-megapixel, 6MB photo, sideways, with EXIF cruft. Suddenly each request carries an order of magnitude more image tokens, prefill time triples, the per-request cost the finance model promised is off by 5x, and a chunk of uploads time out at the load balancer before the model even starts. Nothing in the load test predicted any of it, because the load test used a thumbnail.

The root cause is that multimodal payloads are variable in a way text barely is. A text prompt’s token count is tightly bounded, a chat message is maybe a few hundred tokens. An image’s token count depends on its dimensions and the detail mode, and can span two orders of magnitude between a thumbnail and a full-resolution upload. Audio scales linearly with duration. Your system has to survive the distribution of payloads users actually send, not the median, and definitely not the fixture.

How image tokens actually price out

The pricing is not “an image is one token”, it’s a function of resolution, and it’s worth internalizing because it drives both cost and latency. OpenAI’s vision model tiles high-detail images: scale to fit 2048×2048, then to a 768px short side, divide into 512×512 tiles, and charge 85 base tokens + 170 per tile. A 1024×1024 image is 4 tiles → 85 + 680 = 765 tokens. A wide panorama can be far more. Low-detail mode is a flat 85 tokens regardless of size, a 9x difference from one boolean flag, per the OpenAI vision guide.

Other providers price differently, which matters if you load-test against one and deploy against another. Claude approximates image tokens as roughly (width × height) / 750, a 1000×1000 image ≈ 1,333 tokens. Gemini tiles too, with small images at a flat rate and larger ones split. The takeaway: the same uploaded photo can cost wildly different amounts on different models, so the cost dimension of your load test is model-specific and must use the model you ship.

Audio and video are linear time bombs

Audio is simpler and scarier: it scales with duration. Gemini tokenizes audio at ~32 tokens/second, so a one-minute clip is ~1,920 tokens and a ten-minute meeting recording is ~19,200, before any text. Video is worse still, sampled into frames that each price like images plus an audio track. A feature that accepts “a voice note” needs a load test with voice notes that are actually long, because the user who uploads a 20-minute recording exists, and they will find your timeout.

Three models that all break at once

Big payloads don’t break one thing, they break three, and a text-calibrated test catches none of them:

• The cost model. If finance signed off on “$X per request” based on text-only or thumbnail testing, real image and audio payloads can blow that by 5-10x. The per-conversation economics, which we dig into in cost per conversation under load, are dominated by the largest payloads, not the average.
• The latency model. Image tokens are prefill tokens, and prefill is where time-to-first-token lives. More tiles means a longer blank screen before the first output token. A test that measures TTFT on text prompts will under-predict multimodal TTFT badly.
• The transport layer. Large uploads stress everything before the model: request body size limits, load-balancer and gateway timeouts, multipart parsing, and memory pressure from buffering big files. Plenty of multimodal failures are 413s and 504s at the edge, the model never even runs.

Building a payload-aware load test

The discipline is to drive load with a realistic distribution of payload sizes and to record cost and latency bucketed by that size, so you can see exactly where each model breaks. Start by computing the token cost of each payload so the test knows what it’s sending:

def image_tokens_openai(width, height, detail="high"):
    if detail == "low":
        return 85
    # scale to fit 2048x2048, then shortest side to 768px
    w, h = width, height
    if max(w, h) > 2048:
        scale = 2048 / max(w, h); w, h = w * scale, h * scale
    if min(w, h) > 768:
        scale = 768 / min(w, h); w, h = w * scale, h * scale
    import math
    tiles = math.ceil(w / 512) * math.ceil(h / 512)
    return 85 + 170 * tiles

def audio_tokens_gemini(seconds):
    return round(seconds * 32)      # ~1,920 tokens per minute

# A realistic distribution, NOT a single fixture:
PAYLOADS = [
    ("thumbnail",  image_tokens_openai(256,256)),     # 85
    ("phone_photo", image_tokens_openai(3024,4032)),  # full-res upload
    ("panorama",   image_tokens_openai(8000,2000)),   # the cost spike
    ("voice_30s",  audio_tokens_gemini(30)),
    ("meeting_10m", audio_tokens_gemini(600)),         # 19,200 tokens
]

Now fire requests across that distribution and bucket the results, so a single panorama doesn’t hide inside a median full of thumbnails:

async def multimodal_load(client, payloads, weights, concurrency, n):
    import random, time, collections
    by_bucket = collections.defaultdict(lambda: {"ttft": [], "cost": [], "fail": 0})
    async def one():
        name, tokens, blob = random.choices(payloads, weights=weights)[0]
        t0 = time.perf_counter()
        try:
            ttft = await stream_first_token(client, blob)   # measure prefill latency
            by_bucket[name]["ttft"].append(ttft)
            by_bucket[name]["cost"].append(tokens / 1e6 * PRICE_PER_MTOK)
        except TransportError:                              # 413 / 504 at the edge
            by_bucket[name]["fail"] += 1
    await gather_with_concurrency(concurrency, [one() for _ in range(n)])
    for name, s in by_bucket.items():
        p95 = percentile(s["ttft"], 95) if s["ttft"] else None
        print(f"{name:12s}  ttft_p95={p95}  avg_cost=${mean(s['cost']):.4f}  fails={s['fail']}")
    return by_bucket

The provider-portability trap

There’s a second-order failure that bites teams who treat multimodal endpoints as interchangeable. Because every provider tokenizes images differently, OpenAI at 85 + 170/tile, Claude at roughly (w×h)/750, Gemini with its own tiling, the same uploaded photo can cost two or three times as much on one model as another, and the latency profile differs too. A cost model and a set of timeouts validated against one provider can be silently wrong the moment you add a fallback model or switch primaries for a price cut. The multi-provider failover you set up for resilience, covered in multi-provider failover load testing, can quietly double your multimodal bill on the failover path if you never re-ran the payload load test against the second provider.

The discipline is to run the bucketed payload test against every model your traffic can route to, not just the primary. The numbers that come back will rarely match. A panorama that’s 1,100 tokens on one model might be 1,600 on another; an audio clip that streams its first token in 1.2s on one might take 3s on another. If your timeouts and cost alerts were tuned to the cheaper, faster model, the failover path becomes a latent incident waiting for the day your primary goes down and every request reroutes to the model you never load-tested.

Defenses you can validate with the test

Once the test exposes the cliffs, the mitigations become testable rather than hopeful. Client-side downscaling before upload, resize a 12MP photo to the 768px short side the model will downscale to anyway, cuts tokens, cost, latency, and transport stress in one move, and your load test should prove the savings. Detail-mode selection (low vs. high) is a 9x cost lever for tasks that don’t need fine detail. Explicit payload-size limits and timeouts tuned to your measured prefill curve stop the 20-minute audio upload from silently consuming a 504. Re-run the bucketed test after each change and confirm the spike bucket moved.

Gate the spike bucket in CI

# CI: large-payload requests must stay inside budget
- name: multimodal payload gate
  run: |
    python mm_loadtest.py --concurrency 32 \
      --bucket panorama   --max-ttft-p95-ms 4000 --max-cost 0.05 \
      --bucket meeting_10m --max-transport-fail-rate 0.0
    #   fails if big payloads breach TTFT, cost, or start timing out at the edge

Doing this against real, authenticated endpoints with real files is awkward with script-based tools, you’re wrangling multipart uploads, auth, and CORS. Driving it from the browser, where uploads, session auth, and streaming all already work, is the natural fit and the gap Hit targets: send real images and audio at production concurrency against your real endpoint, and read TTFT, cost, and transport failures bucketed by payload size.

The bottom line

Multimodal is where small assumptions go to die. A photo isn’t a few tokens, it’s hundreds to thousands; a voice note isn’t a sentence, it’s tens of thousands; and the request that breaks your timeout is the one your fixtures never contained. Test with the payload distribution users actually send, bucket cost and latency by size so the spikes don’t hide in the average, and gate the large-payload bucket in CI. The teams that do this stop being surprised by the 5x cost overrun and the 504s that only big uploads trigger.