Elevate the constraint: validation in the AI era

The queue nobody budgeted for

A team rolled out AI coding assistants. Within a month, pull requests had doubled, then tripled. The review queue, the QA cycle, and the release checks stayed as they were, so lead time held flat and then got worse. By the end of the quarter, leadership was asking why the AI investment had not shown up in the delivery numbers.

Writing code was never the only constraint on delivery, but it was the one most teams were tuned around. AI cut it to a fraction. Theory of constraints says throughput is set by the slowest step, and speeding up any other step builds inventory in front of it. The inventory here is unvalidated change. The textbook gives two responses to a constraint: subordinate everything to it, or elevate it. Subordinating here means throttling generation, which hands back the gain you paid for. Elevating means investing in validation until it stops being the constraint.

An agent can produce a plausible change in four minutes. If confirming it is right takes four days, you have not accelerated anything. You have moved the wait.

Faced with a pile of unvalidated change, teams do one of two things. The first is to leave validation as it is. The queue grows, engineers go back to waiting on review, someone blames the tooling, and the organization gets none of the acceleration it paid for. The second is to thin validation to keep pace: shallower reviews, skipped suites, acceptance by signature. This is the tempting one, because for a quarter it looks like progress. Then deliverables become unstable, and the cost shows up as incidents instead of lead time. Both are ways of not elevating the constraint.

The only way through is validation fast enough to keep pace with generation. It has to be fast in both kinds: the checks a machine can run and the checks that need judgment. And fast means different things at different tiers.

Two kinds of validation

A deterministic check gives the same verdict for the same input, every run. Type checks, lint, unit and integration tests, contract tests, builds, migration dry runs, and exact-match visual snapshots all qualify. A machine, including the agent, can run them with no one watching, and a pass means something. Their weakness is that they catch only what someone thought to encode.

A non-deterministic check can give a different verdict between runs or between judges. Three kinds matter here: an AI review of a change against its spec, an eval of a feature that contains model calls, and a human reviewing or accepting the work. Each applies judgment rather than a rule, so each can catch what nobody encoded. Each is slow and expensive, and a single pass or fail means little on its own. They need sampling, thresholds, and a record of judgments over time.

Deterministic checks are already fast on most teams, or can be made fast with known techniques, so the shortfall shows up in the non-deterministic ones. Teams treat those as slow by nature and push them to the end of the process, and that is how the queue forms. Thinning them is how teams fall into the second failure, because those are the checks that catch what the AI got wrong in a plausible-looking way.

One principle follows: judgment is scarce. Every non-deterministic check should have a deterministic check in front of it that filters out everything a machine could have caught. Don't spend judgment on type errors.

Three tiers, two columns

In the agent loop, the agent itself is waiting. The deterministic checks run from one command, documented where the agent will read it, and cost little enough to run after every edit, so the agent runs them unprompted and self-corrects. A slow suite here is the first fix a team should make. The non-deterministic check is a self-review against the spec before the agent declares done. It is cheap, it catches much of "technically works, isn't what was asked," and its verdict goes in the PR so a human can disagree with it later.

At the pull request, the author and the reviewer are waiting. Machine checks come back in under twenty minutes and the whole tier in under an hour, with no human in the critical path except final review. Dependency and security scanning is specific to AI-written code, because agents invent package names and pull in unvetted ones. AI review goes first, commenting like a reviewer and treated like one rather than as a gate. Features with model calls get an eval against a fixed set with a threshold. Human review happens last, on a change that has already been filtered, with the AI's findings in front of it.

Then route by risk. The AI review's output, plus signals the machine already has (files touched, whether a migration, auth path, or public interface is involved, test coverage of the diff), sorts each change into a quick lane or a careful review. Low blast radius and a clean AI review go to the quick lane, where a human confirms the findings in a minute or two. The quick lane is only as good as the AI review's miss rate, so now and then pull a quick-lane change into careful review. This is where the review queue from the opening dissolves. Per-review cost drops because the human is deciding, not discovering, and review count drops because most volume goes through the quick lane. Without routing, tripled PR volume still means tripled reviews, and a skeptical leader will do that math.

At the release, users and the business are waiting. Smoke tests, data and config checks, and a rollback drill run against the release build. The full eval set reruns to catch regressions. Exploratory testing and acceptance are scoped to taste and intent, with each PR's AI review record on hand so testers know where to look, because the earlier tiers settled everything mechanical.

Each tier filters for the next. A check that fails often at release is a signal to build a cheaper version of it a tier earlier. Dogfooding sits outside the tiers: it runs on whatever is deployed, it is not a gate, and its findings are the main source of new checks to push down. A bug found in dogfooding becomes a test or a review criterion, not a release checklist item.

What the team did, and what to ask

The team from the opening fixed it in roughly this order: a one-command check in the agent loop, AI review ahead of human review, an eval for the model-backed feature, then the quick lane. Reviewers confirmed findings instead of discovering them and saved their careful reading for changes that touched auth, data, or a public interface. Release acceptance shrank to the questions only people could answer: whether the feature felt right, and whether it was what they meant to build. Lead time started coming down within a couple of months. The queue didn't vanish. It moved to where judgment was needed.

If you lead an engineering organization, there are four questions to ask your teams.

1. How long does it take to know a change is right at each tier?
2. Which checks are humans doing that a machine could filter first?
3. Where are we spending judgment on things that don't need it?
4. What share of our changes actually need a careful human read, and do we know which ones?

Add the answers up and they look like a quarter of engineering time on test infrastructure, which is where most plans stall. The estimate is wrong, because the gates are code and the agents write code. The missing tests, the fast suite, the eval set, the review instructions: an agent builds each of them in hours, and the scarce judgment goes into checking what it built. The next post shows how, with four examples from my own work.

AI made producing change cheap. Teams now choose between validating fast, validating slow, or not validating. Only one of those is acceleration.