The QA Engineer's Survival Guide to LLMs

You've spent your career testing deterministic systems. Press button, get expected output. Test it three times, get the same result. That era is over.

LLMs have arrived, and they break every assumption you've built your QA career on. But here's the good news: you don't need to become a machine learning engineer. You need to understand what's different, and then adapt the testing skills you already have.

This is your field guide to LLM testing.

What You Already Know Still Works (Sort Of)

Your traditional QA skills are still valuable. Test cases still exist. Regression testing still matters. Boundary testing still applies. You can still use mocking, fixtures, and automated test runs.

The difference: in deterministic systems, passing a test case once meant it was solved. In LLM systems, passing 100 times out of 100 and failing 1 time is normal. Your confidence interval just changed, not your methodology.

What transfers directly:

• Test design (happy path, edge cases, error states)
• Test automation (scripting, CI/CD integration)
• Regression testing (comparing outputs across versions)
• Exploratory testing (finding where the system breaks)
• Documentation and reproducibility

What's completely new:

• Statistical measures instead of binary pass/fail
• Aggregate patterns instead of individual outputs
• Calibrating confidence thresholds instead of expecting perfection

Tokens: The Currency of LLM Testing

An LLM doesn't think in words. It thinks in tokens, small chunks of text, roughly 4 characters each. Understanding this changes how you test.

One token is approximately:

• 4 characters in English
• 1-2 words (varies widely)
• 1 number (usually, sometimes more)

Why does this matter for testing? Because:

• Cost scales with tokens. A prompt that's 1,000 tokens costs roughly 1,000x more than 1 token (oversimplified, but directionally right)
• Speed depends on tokens. More tokens = slower responses = timeout risks in your integration tests
• Accuracy changes at token boundaries. The model processes text differently depending on how it tokenizes your input
• Context windows limit input size. Claude Opus has a 200K token context window, roughly 150K words. That sounds infinite until you're testing with long documents

For QA, this means:

TEST: Token counting accuracy
INPUT: "What is AI?"
TOKENS: 4 (usually)

INPUT: "What is artificial intelligence?"
TOKENS: 6

TEST RESULT: Different token counts =
potential different model behavior
even though inputs are semantically similar

IMPLICATION FOR TESTING:
- Test both short and long versions of prompts
- Verify cost estimates account for actual tokenization
- Test near context window boundaries

Temperature: The Chaos Knob

Temperature controls how random the model is. If you've never heard of it, this is the single most important LLM parameter for QA.

• Temperature 0.0: Deterministic (mostly). Same input, same output, every time. Use this for testing.
• Temperature 0.5: Balanced. Consistent but with subtle variation. Use this for production systems that need reliability with slight randomness.
• Temperature 1.0: Default chaos. High creativity, unpredictable. Each response is different. Don't use for testing unless you're specifically testing variability.
• Temperature 2.0+: Complete randomness. Outputs barely relate to inputs. Only for experiments, never for production.

temperature affects your test cases:

SYSTEM_PROMPT = "You are a helpful assistant."
USER_PROMPT = "What is 2+2?"

WITH TEMP=0.0:
Response 1: "2 + 2 = 4"
Response 2: "2 + 2 = 4"
Response 3: "2 + 2 = 4"
TEST PASSES (100% consistency)

WITH TEMP=1.0:
Response 1: "The sum of 2 and 2 is 4."
Response 2: "Two plus two equals 4."
Response 3: "When you add 2 to 2, you get 4."
TEST FAILS (same answer, different text)
OR TEST REQUIRES fuzzy matching

"Temperature is your debug knob. Struggling with flaky tests? Lower the temperature. Need to verify the model is creative? Raise it. But never set it once and forget it."

Context Windows: Your Hard Ceiling

A context window is the maximum amount of text (in tokens) that you can feed into the model, plus the response it generates.

Claude Opus has 200K tokens. GPT-4 has 128K. That sounds enormous until you're testing:

• A system that summarizes 500 legal documents (each 1K tokens)
• A chatbot that remembers conversation history across 50 messages
• An AI that needs to reference multiple code files plus your question

Your testing needs to include context window boundary tests:

TEST_CASE: Context Window Boundaries
SETUP: Progressive input size increase

RUN 1: Input 10K tokens → Expected: Completes
RUN 2: Input 50K tokens → Expected: Completes
RUN 3: Input 100K tokens → Expected: Completes
RUN 4: Input 150K tokens → Expected: Completes
RUN 5: Input 180K tokens → Expected: Completes
RUN 6: Input 200K tokens → Expected: Complete or graceful failure?
RUN 7: Input 210K tokens → Expected: Graceful error message

VALIDATION:
- At what size does performance degrade?
- At what size does it fail?
- Does it drop data from beginning or middle?
- What's the error message?
- Did you build a circuit breaker to refuse > 200K?

This is basic load testing, but for text instead of requests. Test it before your users hit these limits.

Embeddings: The Semantic Fingerprint

Embeddings convert text into a list of numbers (usually 1000+ of them) that represent meaning. Two sentences with similar meaning have similar embeddings, even if they use different words.

For QA, embeddings matter because:

• Similarity matching: You can test whether the model's response is semantically similar to expected output, not just text-identical
• Hallucination detection: Compare embeddings of generated text with source documents to detect when the model invents information
• Clustering: Group model outputs by semantic meaning to find failure patterns

A practical test case using embeddings:

TEST: Answer semantic similarity
EXPECTED_ANSWER = "The capital of France is Paris."
MODEL_OUTPUT = "Paris is the capital city of France."

TRADITIONAL TEST: FAIL (different words)

EMBEDDING-BASED TEST:
embedding_expected = vectorize(EXPECTED_ANSWER) # [0.21,0.88,0.45, ...]
embedding_output = vectorize(MODEL_OUTPUT) # [0.22,0.87,0.44, ...]

similarity = cosine_similarity(embedding_expected, embedding_output)
# Result: 0.987 (very high, >0.95 = semantically equivalent)

TEST: PASS (98.7% semantic match)

This is how you move from brittle string matching to robust semantic testing.

Prompts: Your Test Cases Are Now Instructions

In traditional software, test cases are independent of each other. In LLM systems, the prompt IS the test case, and it can be devilishly complex.

A well-designed test prompt for an LLM includes:

SYSTEM_PROMPT = """
You are a customer support assistant.
Your tone is professional but friendly.
If you don't know the answer, say so.
Never make up product features.
Cite sources when providing information.
"""

USER_PROMPT = """
I bought product X three months ago and it stopped working.
What should I do?
"""

CONSTRAINTS = {
 "max_tokens": 300,
 "temperature": 0.5,
 "top_p": 0.9,
 "stop_sequences": ["Human:", "Assistant:"]
}

EXPECTED_BEHAVIOR = [
 "Acknowledges the problem",
 "Offers troubleshooting steps",
 "Mentions warranty/return policy",
 "Does NOT admit fault (legal concern)",
 "Suggests escalation if unresolved"
]

Notice: the test case includes system prompts, user input, model parameters, AND expected behaviors (not exact outputs).

How Your Testing Skills Actually Transfer

Boundary testing: Instead of testing input value ranges, test semantic edge cases. "What if the prompt is ambiguous?" "What if the query contains contradictions?"

Regression testing: Instead of checking exact outputs, compare embedding-based similarity or semantic correctness across model versions.

Load testing: Test context window usage, token count scaling, and latency as you increase input complexity.

Error handling: Test what happens when the model can't complete a request, when the context window is full, when rate limits are hit.

Documentation: Document your test prompts obsessively. They're the only reproducible record of what you tested.

What's Genuinely New

Statistical testing: You need to think in confidence intervals. "This feature works 97% of the time" is not the same as "works sometimes." Run 100+ samples to validate thresholds.

Prompt engineering as test design: Small changes to prompts cause massive output changes. Your test design is now partly about finding the right prompt structure.

Hallucination testing: Verify that the model doesn't invent information. Cross-reference outputs with source documents. This is unique to LLMs.

Bias and fairness audits: Test that the model behaves fairly across demographics, languages, and cultural contexts. This is testing in a new dimension.

"Your biggest skill advantage right now is that you're a QA engineer, not an ML engineer. You already think like a tester. Just swap 'deterministic' for 'probabilistic' and you're most of the way there."

The Three-Level Testing Pyramid for LLMs

Level 1: Unit tests (80% of tests)

Test individual prompts with fixed temperature, controlled inputs, semantic similarity matching. These are fast and catch obvious failures.

Level 2: Integration tests (15% of tests)

Test the full pipeline: user input → prompt construction → model call → output parsing → response delivery. Test context passing between components.

Level 3: End-to-end tests (5% of tests)

Test real user scenarios with real data. These are slow but catch integration failures you can't simulate. Run these weekly, not per-commit.

Tools You Actually Need

• Prompt testing: Langfuse, OpenAI Evals, or homemade test harness in Python
• Semantic matching: OpenAI embeddings or open-source alternatives
• Monitoring: alt.qa, LangSmith, or custom dashboards tracking semantic correctness over time
• CI/CD: GitHub Actions or Jenkins running your LLM tests, same tools, new logic

The Paradigm Shift

Stop thinking "Does this work?" Start thinking "How often does this work, and under what conditions?"

You're not testing for existence anymore. You're testing for reliability at scale, semantic correctness, and graceful failure. The tools change, but the testing mindset is exactly what you've been doing for years.

You've got this. The LLM world just got a QA team.