Keita Broadwater’s Reliability Engineering for AI Systems, a free book still being written in public, argues that an AI agent’s dependability is a probability you have to measure under stated conditions, not a feeling you get from a good demo. Visa’s new shopping-agent benchmark shows why that matters, and where the book’s published chapters stop short.
On 5 November, Visa published what it called a first-of-its-kind benchmark for AI shopping agents. It tested 22 models from 12 labs on 15 shopping tasks, such as choosing a TV whose buyer has not yet said what they need. According to Visa’s announcement, the models passed 57.2% of runs, the best 82.2% and the worst 28.9%. Every model got three attempts at each scenario, and all three attempts passed just 34.2% of the time. About 54% of failed runs still ended in a purchase. Visa’s own summary of the problem is blunt: “when made to repeat a scenario they had already passed, the same models often failed.”
That sentence is the whole subject of Reliability Engineering for AI Systems: Measuring, Predicting, and Controlling Failure in LLMs, Agents, and Their Harnesses, an early-access book by Keita Broadwater from SafeFlow Press. It is free to read online without an account. This review is based on the four chapters and one appendix published in version 0.4.0, revised on 7 November 2026, close to 39,000 words including code listings. The book’s own contents page lists 21 planned chapters, so what follows is a judgment on the first fifth of the book, not on a finished work.
What the book argues
The opening line sets the tone: “An AI system does not have to crash to fail.” Chapter 1 follows Dana, a customer whose bank chatbot tells her, repeatedly and politely, that it has opened a dispute over an $842 charge she does not recognize. It never does, because the tool call behind the confirmation keeps failing. Broadwater says the example is adapted from complaints in the US Consumer Financial Protection Bureau’s complaint database. It works because every metric a bank might watch (availability, response time, fluency) would have marked her conversations a success.
From there the book makes one argument and makes it carefully. Reliability, borrowed in its strict engineering sense, is the ability of a system to perform its intended function under specified conditions over a specified amount of use. For a large language model (LLM) or agent, that means three things. You must define failure before you measure it. You must test the whole system, meaning the model plus the prompts, tools, parsers and services around it, not the model alone. And you must run it many times, because a single run tells you almost nothing about the probability of the next one.
Broadwater writes from hardware. He describes years spent in laboratories with temperature and humidity chambers, testing motherboards, processors and semiconductors, and he maps the bare silicon die and its package onto the model and its surrounding application. He is careful with the comparison: “The analogy between an LLM and a semiconductor chip is not perfect, and I do not want to push it too far.” The borrowed discipline is the useful part. Hardware engineers did not test one chip and declare a design reliable. They tested populations, counted failures and modeled them.
How it is built
The published chapters climb one ladder. Chapter 1 sets out the mental model and vocabulary: failure, failure mode, system boundary, operating condition. Chapter 2 turns repeated execution into an experiment, a “reliability harness” that resets state, runs a trial, collects evidence, measures the outcome and keeps the record. It uses a small simulation of Dana’s bank so readers can run it without calling any external service. Chapter 3 moves to real models and statistics. Chapter 4 turns the lens on the thing doing the grading, the LLM judge. The chapters tie to a companion GitHub repository under an MIT license, with raw responses, manifests and scripts that regenerate the figures.
The prose is plain and patient. Equations appear only when they earn their place, and each chapter closes with a short “reliability principle” box. A reader who last met a confidence interval in a university statistics course can follow it.
What it gets right
The denominator is an experiment, not a score
Chapter 2’s best passage is about when repeated runs can be pooled. Suppose a bank runs Dana’s scenario 50 times with one prompt and sees one failure, then 50 times with another prompt and sees nine. The pooled 10% is correct arithmetic and may describe neither system. Broadwater insists that the usual statistical assumption, that trials are independent and identically distributed, “should therefore be treated as an experimental assumption, not as a property we automatically acquire by running the same test repeatedly.” He adds that “thirty failures produced by one outage do not necessarily provide the same statistical information as 30 failures arising separately across otherwise independent executions.”
Real numbers, honestly bounded
Chapter 3 reports a confirmatory experiment of 1,500 runs: 500 each for three small local models (Qwen2.5 0.5B, Llama 3.2 1B and Phi-3.5 3.8B) on a fixed banking decision that must be returned as a strict JSON object. Qwen failed 484 times, Llama 55 and Phi 50. Broadwater then shows why a one-point gap between 11% and 10% is not a ranking. The 95% confidence interval for the difference runs from about −2.83 to 4.84 percentage points. Because that range includes zero, the data do not establish which model is more reliable. His model sentence for a reliability claim is worth copying into any evaluation report. It names the configuration, the task, the 50 failures in 500 valid runs, and a Wilson interval of 7.67% to 12.94%, and it says the estimate applies only to that condition.
The chapter’s zero-failure section is the one most product teams need. “Zero observed failures does not establish zero failure probability,” the book states, and it introduces the rule of three: after N clean runs, the 95% upper bound on failure is roughly 3/N. To show a failure rate below 0.1% with no failures allowed, you need at least 2,995 clean trials. Broadwater adds the uncomfortable corollary that a system which truly fails 0.05% of the time has only about a 22.4% chance of passing that test.
The parser is part of the product
The most striking result in Chapter 3 needs no new model calls. Broadwater re-scores the same 1,500 saved responses with a more forgiving parser that can pull a valid object out of surrounding text. Qwen’s failure rate falls from 96.8% to 38.0% and Phi’s from 10.0% to 3.4%. Llama’s stays at 11.0%, because its failures were wrong decisions, not formatting slips. Same model, same outputs, different system, different reliability. It is a clear demonstration that “model reliability” is often a property of the code around the model.
Judges are instruments, and instruments fail
Chapter 4 is the most original part of the book so far. Broadwater builds an LLM judge to re-grade the 1,500 responses and treats it as a measuring instrument that must qualify before it is trusted. The first attempts are sobering. A hosted model the book calls GPT-6 Luna, run with reasoning disabled, classified 1,495 of 1,500 responses correctly. Under the original specification, local Gemma 3 4B returned 1,500 abstentions, hosted Qwen3.5 9B returned no usable binary judgments, and GPT-OSS 120B returned just 21. In diagnosis, some judge explanations treated text inside the responses being graded as if it added new grading rules, a failure with an obvious security echo. Broadwater is careful to call this a supported hypothesis, not an established mechanism. After revision, GPT-OSS scored 1,497 of 1,500, better than a two-of-three majority vote of the panel.
The chapter refuses easy conclusions. A unanimity rule that sent disagreements to human review handled 97.8% of cases automatically and got all but one right. That one error was a response all three judges accepted and the reference rejected. “Consensus reduced the number of errors among automatically handled cases,” Broadwater writes, “but it did not make agreement equivalent to correctness.” He is equally careful about explanations: they “are observable outputs that help us investigate the problem, not direct evidence of the model’s internal reasoning.”
Reading Visa’s benchmark with the book’s tools
Visa’s benchmark white paper is more careful than most. It grades with a rule-based answer key, reruns runs broken by provider faults, and counts a lucky correct purchase as a failure if the agent never uncovered the buyer’s requirements. It also adopts the passk measure from the τ-bench research benchmark, the chance that k randomly chosen runs of the same task all succeed, and states that “a capability is considered demonstrated only when it appears consistently across runs, that is, when passk stays high as k grows.” Broadwater would agree with every word.
Apply his Chapter 3 arithmetic to the headline numbers, though, and they shrink. Fifteen scenarios at three attempts each is 45 runs per model. The reported best and worst scores, 82.2% and 28.9%, match 37 and 13 passes out of 45 exactly. That is our inference from Visa’s description, assuming no runs were dropped. On that basis, our calculation of a 95% Wilson interval for the top score runs from about 69% to 91%. The worst score’s interval runs from about 18% to 43%.
That matters for the rankings that followed. The Next Web, citing Fortune, reported that agents from Google and Meta tied for the top score at 82%, a Claude agent came third with 76% and an OpenAI agent sixth with 64%. Using the same Newcombe method the book uses, our 95% interval for the gap between first and third place runs from about −10 to +23 percentage points. Even first versus sixth, roughly 37 against 29 passes out of 45, gives an interval from about −0.5 to +35 points, which includes zero. On 45 runs per model, the benchmark separates good shoppers from bad ones. It cannot reliably rank the leaders.
The consistency figure needs the book’s Chapter 2 caution. If every run were an independent coin flip at the pooled 57.2%, three passes in a row would happen about 18.7% of the time. Visa observed 34.2%, nearly double. That is what you would expect when 22 models with very different abilities and 15 scenarios of different difficulty are pooled into one number: some model-and-scenario pairs nearly always pass and some nearly always fail. The pooled 57.2% is a mixture, exactly the kind Broadwater warns can “conceal” separate failure propensities. It does not undercut Visa’s point that agents are inconsistent. It does mean the inconsistency that matters is per model and per task, and three attempts per task are far too few to measure it. By the rule of three, even a model that passed all 45 of its runs could only be said, with 95% confidence, to fail less than about 6.4% of the time. For an agent spending a customer’s money, that is not a strong guarantee.
Chapter 4 also applies. Visa’s white paper says the customer in each conversation “is simulated by a model.” Visa grades the final decision with its rule-based key, so it avoids the LLM-judge problem Broadwater dissects. But the simulated customer is still a model-driven part of the measurement, and the book’s argument is that every such component has to be qualified before its outputs count as evidence. Visa says it plans to calibrate that customer against recorded human shopping sessions. The book suggests that calibration should be reported with the same care as the scores.
What it gets wrong, or has not reached yet
The gap between the subtitle and the published text is wide. The book promises agents and their harnesses. The published experiments use three models of 3.8 billion parameters or fewer, run locally, on one fixed transaction with one correct answer. That is a deliberate choice for isolation and reproducibility, and Broadwater says plainly that his conclusions are limited to that condition. But it means the book has not yet shown its methods working on the multi-step, tool-using, stateful agents that Visa tested and that companies are deploying. Tool-using agents, long-running agents and failure cascades all sit in Part III, marked “planned.”
The repeated-use problem is also deferred. The passk question at the heart of Visa’s report, the chance that a task which worked once keeps working, belongs to a planned chapter on latent failure under repeated inference. The published chapters give you the binomial model and its intervals, but not yet the tools for heterogeneous, condition-dependent failure that would let you take apart a number like 34.2%.
The statistics themselves are standard: Wilson intervals, the rule of three, Newcombe differences and exact binomial bounds. That is a strength for teaching, since practitioners need them and rarely use them. It is not new theory, and readers from biostatistics or quality engineering will find little they have not seen. The book’s originality lies in the experiments and the measurement-layer diagnosis, not the mathematics.
Finally, this is a self-published early-access work with little external validation. The repository had no GitHub stars when we checked, the author biography is brief, and chapters may change, as the site warns. Readers should treat the empirical results as one careful author’s reproducible experiments, which is what they are, and check them against the posted data.
Who it is for
Engineers and product managers who ship LLM features and have been asked “how reliable is it?” should read Chapters 2 and 3 now. Teams that use LLM judges to grade their own systems should read Chapter 4 before trusting another automated eval dashboard. Anyone reading AI benchmark press releases, including journalists, will come away better able to ask how many runs, under what conditions, and with what interval.
It is not yet the book for someone who needs a full method for evaluating production agents, and it is not a policy book. Its claim is narrower and more useful: in the book’s words, “a statistical interval cannot compensate for an incomplete or inappropriate experimental specification.”
The verdict
The first four chapters of Reliability Engineering for AI Systems are a clear, honest and reproducible introduction to measuring how often AI systems fail, and Chapter 4’s treatment of judges as fallible instruments is among the most useful free treatments of the subject available. The book arrives just as the industry has started publishing agent benchmarks with pass-rates and consistency figures, and it supplies exactly the vocabulary needed to read them skeptically. Its weakness is timing: the chapters on agents, the very systems that make the question urgent, are not written yet. The question for Broadwater’s next releases is whether his single-transaction discipline survives contact with a shopping agent that talks, searches and spends. Visa’s benchmark suggests that is where the reliability problem now lives.