Docker for AI Testing: The 2026 Playbook for Reproducible LLM Validation

Containerization is the foundation of reproducible testing. For traditional services, the patterns are well-known: small base image, layered builds, multi-stage compilation. Run it anywhere, get the same result.

For AI testing, every assumption breaks. Models are 30GB instead of 30MB. GPUs need driver coordination across host and container. Floating-point operations on different GPUs produce different results. A "reproducible" CI run with the same Dockerfile can produce different model outputs on different runners.

This guide is the 2026 playbook for actually reproducible AI testing in Docker.

Why naive Docker breaks for AI tests

The classic Docker pattern: FROM python:3.11-slim, pip install, copy code, run tests. Total image size: 200MB. CI run: 2 minutes.

Apply that to an AI testing pipeline that loads a 13B-parameter model, and you get:

• Image size: 28GB (13B params × 2 bytes/param + tokenizers + dependencies)
• First run on a fresh runner: 12-18 minutes (mostly model download)
• Cache miss after a Dockerfile edit: full re-download, full re-build
• "Why is the model output different on my Mac vs CI?", six hours of debugging

The solution is not to "make Docker faster." It is to architect AI tests around the constraints Docker imposes.

Pattern 1: model artifacts as volumes, not image layers

Never bake a model into a Docker image. Models change rarely; code changes constantly. Bake the runtime, mount the model.

# Dockerfile.test
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3-pip
RUN pip install vllm==0.6.3 transformers==4.45 pytest==8.3

WORKDIR /app
COPY src/ ./src/
COPY tests/ ./tests/

# Model is mounted, not copied
ENV MODEL_DIR=/models
ENV HF_HOME=/cache/huggingface

ENTRYPOINT ["pytest", "tests/"]

And run with:

docker run --gpus all \
 -v /shared/models/Llama-3.1-8B:/models:ro \
 -v /shared/cache/hf:/cache/huggingface \
 --shm-size 16g \
 ai-tests:latest

Image stays at ~3GB (CUDA + Python + libs). Model stays on a shared NFS or persistent disk. Cold-start drops from 18 minutes to 2 minutes. This single change is the highest-leverage Docker fix for AI testing.

Pattern 2: GPU runtime layers

Different test suites need different GPU profiles. A unit test that mocks the LLM doesn't need a GPU at all. An integration test that spins up vLLM needs an A10. A regression test against a 70B model needs an H100.

Build three images from a shared base:

# base: shared deps
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04 AS base
RUN pip install transformers pytest

# CPU profile: for unit tests that don't need GPU
FROM base AS cpu
ENV TRANSFORMERS_OFFLINE=1 USE_CPU=1

# Small-GPU profile: for vLLM integration with 7B models
FROM base AS gpu-small
RUN pip install vllm==0.6.3
ENV VLLM_GPU_MEM_UTIL=0.5

# Big-GPU profile: for full eval against 70B
FROM base AS gpu-large
RUN pip install vllm==0.6.3 flash-attn==2.6
ENV VLLM_TENSOR_PARALLEL=8

CI selects the appropriate target with --target cpu, --target gpu-small, etc. Most unit tests run in seconds on a CPU runner; only the slow regression suite needs a GPU runner.

Pattern 3: deterministic kernels

cuDNN selects different convolution algorithms based on input size, GPU model, and available memory. The same Docker image on a T4 vs an A100 produces different floating-point outputs. The same image on an A100 with concurrent jobs produces different outputs than a quiet A100.

For tests where output equality matters, force determinism:

import torch
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.manual_seed(42)
import os
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
os.environ["PYTHONHASHSEED"] = "42"

Even with these flags, expect ~1e-6 level numerical drift across GPU generations. Don't write tests that assert bitwise output equality across hardware. Write tests that assert tolerance.

Pattern 4: caching the HuggingFace download

Without a cache, every CI job re-downloads models from HuggingFace. With a cache, you risk version drift. Solve this with content-addressable mounts:

# In CI: mount a shared, immutable model cache keyed by hash
docker run --gpus all \
 --mount type=bind, src=/cache/hf-models, dst=/cache, readonly \
 -e HF_HOME=/cache \
 -e TRANSFORMERS_OFFLINE=1 \
 ai-tests:latest

Pre-populate the cache during a daily warming job, never during a per-PR test job. This separates the slow operation (downloading) from the fast operation (running tests).

Pattern 5: container-friendly observability

AI tests need richer signals than pass/fail. Capture them inside the container, write them to a mounted volume, surface them in CI:

• Per-test latency distributions (p50/p95/p99)
• GPU memory peak (via nvidia-smi --query-gpu=memory.used)
• Tokens generated, cache hits, KV-cache pressure
• Full prompt/response logs (gzipped, dumped to volume)

The pattern: tests write JSONL events to /artifacts/eval.jsonl mounted at /var/lib/ci/runs/$RUN_ID/. Post-run, a separate container parses, summarizes, and uploads to your eval dashboard.

Common 2026 stack notes

NVIDIA Container Toolkit is required for any GPU access in Docker. Use the runtime nvidia, not the legacy nvidia-docker.

vLLM in containers needs --shm-size set above the default 64MB or you'll see CUDA OOM errors that look like memory issues but are actually shared-memory issues. --shm-size 16g is a safe default.

BuildKit cache mounts (--mount=type=cache) cache pip downloads across builds; combined with model volumes, builds drop from minutes to seconds.

Air-gapped environments (defense, healthcare) ship pre-baked images with model artifacts inside. The 30GB image is the cost of doing business in those settings, but for general CI, never.

The reproducibility checklist

For a Docker-based AI test to be reproducible, all of these must be true:

1. Pinned base image digest (not :latest)
2. Pinned Python and CUDA versions
3. Pinned model hash (HuggingFace revision SHA, not branch name)
4. Deterministic flags enabled (cudnn.deterministic = True, fixed seeds)
5. Tolerance-based assertions, not bitwise equality
6. Same GPU class across runs (test on the same SKU as production)
7. Locked dependency versions (uv lock or pip-tools)

Miss any one of these and your tests will pass on a Tuesday and fail on a Wednesday for reasons no one can explain.

Containers are not magic, they are a contract. The contract is: same inputs, same outputs. For AI testing, that contract requires deliberate engineering at every layer: image, runtime, hardware, kernel, model. Get those right and your CI becomes the most reliable part of your AI stack.