Engineering enablement · Review and release

Review and release controls that catch what generated code gets wrong

If generated code is swelling your review queue and you need quality controls that do not rely on a model saying the code looks fine, you can commission a review and release programme. I redesign the review workflow by risk, run a seeded-defect evaluation to measure what your reviewers, AI reviewers and tests actually catch, and put release acceptance controls into CI.

This is a good fit if…

  • Pull requests have grown since AI coding tools arrived, and senior engineers spend much of their week reviewing.
  • You have switched on an AI code reviewer and nobody knows how many real defects it finds or misses.
  • Defects are reaching production that tests should have caught, and generated tests often assert whatever the code already does.
  • You need release controls you can explain to an auditor or customer, not a reviewer's good intentions.

Look elsewhere if…

  • Your team has not yet agreed how it uses AI tools at all. Start with AI engineering enablement for software teams.
  • You have an AI-generated application that needs to be made production-ready. Use take an AI-generated application into production instead.
  • You need to evaluate an LLM feature's answers rather than code changes. Use production LLM evaluation instead.

What you get

Review workflow, seeded-defect evaluation and release acceptance controls

  • A review workflow that routes changes by risk, so senior attention goes where it matters.
  • A seeded-defect evaluation showing what your human reviewers, AI reviewer and test suite each catch and miss.
  • Release acceptance controls in CI: required checks, test-quality checks on changed code, size limits and staged release for risky paths.
  • A rule, written and enforced, that an AI review never counts as an approval.
  • Baseline and post-change measures of review time, review rounds and escaped defects.

How it runs

  1. 01

    Baseline review and release

    From your pull-request and incident data, we measure review time, rounds, pull-request size, reverts and escaped defects, and map how changes reach production.

  2. 02

    Seeded-defect evaluation

    In a safe copy of the repository, we plant known defects in realistic changes and measure what reviewers, the AI reviewer and the test suite catch.

  3. 03

    Redesign review and release

    Using the results, we set risk tiers, add CI controls where the evaluation found gaps, and agree the review rules with the team.

  4. 04

    Run, measure and hand over

    The team works under the new workflow for at least two full sprint cycles; we re-measure and the engineering manager takes ownership.

What needs to be in place

  • An engineering manager or developer experience lead who owns the review process.
  • A repository with CI where controls can be added, and admin access to repository rules.
  • Pull-request, CI and incident data for a recent period.
  • Time from several reviewers to take part in the seeded-defect evaluation.

Not included

  • Treating any AI reviewer's approval as proof of correctness or as a required approval.
  • Security penetration testing or a formal security audit.
  • Guaranteed defect reduction. The programme measures what changed.
  • Ranking or performance-managing individual reviewers from evaluation results.

Why review is now the bottleneck

AI coding tools made writing code cheaper. They did not make reading it cheaper. Teams that adopted assistants and agents often find that pull requests got larger and more frequent, while the number of people who can judge them stayed the same. Senior engineers spend their days reviewing, and some of what they approve contains defects that a smaller change would have made obvious.

Two tempting answers make it worse. One is to approve faster, trusting that generated code is probably fine. The other is to let an AI reviewer carry the load, as if a model saying “looks good” were evidence of correctness. A model’s review is another opinion. It is useful, but it proves nothing.

This programme puts the evidence somewhere else: in measured detection rates, in tests that fail when behaviour breaks, and in release controls that do not depend on anyone’s vigilance.

What the work involves

Baseline. We measure review and release as they are: pull-request size, time to first review, review rounds, reverts, change-failure rate and defects found after release, and map every route a change takes to production.

Seeded-defect evaluation. In a safe copy of the repository, we take realistic changes and plant known defects of the kinds that matter for your code: logic errors, missing authorisation checks, unhandled failures, concurrency mistakes, and tests that assert the wrong behaviour. We then measure which defects your human reviewers, your AI reviewer and your test suite catch. The result is a detection map for your codebase, not a vendor’s benchmark.

Risk-tiered review. Using the map, we set review tiers. Low-risk changes with strong tests get a lighter review. Changes to authentication, payments, data migrations or shared infrastructure get senior review and smaller size limits. AI-assisted changes are labelled, and their author owns them.

Release acceptance controls. Where the evaluation found gaps, we add controls to CI: required checks on every merge, test-quality checks on changed code (such as mutation testing), pull-request size limits, and staged or flagged release for risky paths with a rehearsed rollback.

This is the method I have published as Vibes Inside Guardrails: rely on mechanical constraints (tests, types, CI gates) rather than on reviewers catching everything by eye.

The signature deliverable

You end with a review workflow, a seeded-defect evaluation and release acceptance controls. Illustrative example of an evaluation summary:

Seeded defect typeHuman reviewAI reviewerTest suiteControl added
Off-by-one in paginationcaught / missedcaught / missedcaught / missedProperty test on changed module
Missing authorisation checkcaught / missedcaught / missedcaught / missedSenior review tier for auth paths
Swallowed errorcaught / missedcaught / missedcaught / missedLint rule in CI
Test asserts current wrong outputcaught / missedcaught / missedn/aMutation testing on changed files

Illustrative example. The format; results come from your own evaluation.

How acceptance is judged

The engineering manager accepts the programme against the baseline: review time and rounds, pull-request size, escaped defects and reverts, measured again after at least two full sprint cycles under the new workflow. The seeded-defect evaluation can be re-run to check that detection improved where controls were added.

Ownership and handover

The review tiers, CI controls and evaluation kit belong to your team. The kit is documented so you can re-run the evaluation after a tool or process change. The rule that AI review is advisory is written into the repository rules and the contributing guide.

Boundaries

If the team has not yet agreed how it uses AI tools, start with AI engineering enablement for software teams. For rules about what coding agents may do, see coding-agent rollout. For Copilot’s own review features, see GitHub Copilot enablement. To make a whole AI-generated application production-ready, see take an AI-generated application into production.

Questions buyers ask

What is a seeded-defect evaluation?

We take realistic changes and deliberately insert known defects: an off-by-one, a missing permission check, an unhandled error, a test that asserts the wrong thing. Then we see which ones reviewers, the AI reviewer and the tests catch. It turns 'the AI reviewer seems useful' into a measured detection rate for your code.

Won't reviewers feel tested?

The evaluation measures the review system, not individuals, and results are reported in aggregate. Reviewers take part knowingly, and the usual finding is that defects slip through because changes are too large or tests are weak, which are process problems.

Should we turn off our AI reviewer?

Usually not. As a first pass it can catch some issues before a person spends time. The evaluation shows which kinds of defect it finds and which it misses, so reviewers know where to focus. What changes is its status: it comments, it never approves.

How do you stop generated tests that just mirror the code?

Release controls check test quality on changed code, not only coverage: tests must fail when the behaviour they cover is broken. Mutation testing on changed files, or seeded checks, can show whether a test suite would notice a defect.

Describe what needs to work

A short, non-confidential description is enough to start. I read every brief personally and reply within two business days, including when the answer is that I am not the right fit.

Step 1 of 2 · The basics