How a cell is run, and how pass or fail is decided.
A skeptical QE lead should be able to predict how any cell will be judged from this page alone.
Every agent runs the same observe → decide → act loop, logged identically.
Observe
Screenshot, DOM snapshot hash, accessibility tree, console and network state are captured before every action.
Decide
The agent (Claude, Gemini, Browser Use…) proposes the next action toward the goal. Persona parameters constrain what it may do: a keyboard-only persona cannot click; a 44 px dexterity persona cannot hit smaller targets; a B1 reader is given text at its reading level.
Act
The action runs in a real browser under alt.qa’s signed identity. Irreversible actions and payment steps are blocked unless the Test Authorization allows them in a sandbox.
Judge
The goal’s success criteria (for example “reached payment page with 2 nights selected”) are checked against the final state, not the agent’s own opinion of success.
Three levels. Only one becomes a ticket.
| Level | Rule | Ticketed? |
|---|---|---|
| Confirmed | Reproduced in ≥ 2 cells or by deterministic replay | ✓ Yes |
| Likely | One cell, consistent trace | ✕ Review queue |
| Reported | Persona narrative only | ✕ Never |
Reading the matrix
- Failing column → an agent-class problem (e.g. bot management blocks one vendor).
- Failing row → a persona problem (e.g. targets too small for low dexterity).
- Same step failing in ≥ 3 cells → a product bug, ranked highest.
Each finding is tagged with one of 16 failure classes: from bot management, CAPTCHA and consent walls to focus traps, colour-only signals, stale prices and missing confirmations before irreversible actions.
Reproducibility
Cells can be repeated 1-5 times with fixed seeds; variance across repeats is reported. Model version strings and dates are stored on every cell. DOM snapshots are cached so a finding can be re-scored without re-running the agent.
Behavioural approximations
Some columns approximate how a class of consumer agent behaves (pacing, how it asks for confirmation) using a vendor model under our own identity. They are labelled as approximations everywhere and never spoof a real product’s identity.
Calibration
Paid testers from the populations we model re-run Confirmed findings. Their verdicts produce the published precision. This follows the “validate, then calibrate” approach recommended for LLM-simulated users.
Personas are simulations, not people. Findings are evidence of behaviour we observed on a given date with a given model version. They are never a statement that a site is “compliant”, “certified” or “agent-ready”. Consumer-agent classes are behavioural approximations, not the named products. Every published precision figure states its sample size and confidence interval. Read the full limitations statement →