The Copilot question most organisations cannot answer
GitHub Copilot is often the first AI coding tool an organisation buys, because it sits inside tools developers already use and is administered alongside GitHub itself. A year later, the usage dashboard shows active users and suggestion acceptance. What it cannot show is whether pull requests got bigger, whether reviewers are spending longer, or whether more defects reach production. Meanwhile Copilot has grown from completions and chat into code review and an autonomous cloud agent, each with its own settings.
This pilot answers the question that matters to a developer experience lead: on this repository, with this team, does Copilot improve delivery without adding review or security debt?
I work independently of GitHub and Microsoft: there is no affiliation, endorsement or vendor certification. The product details below are checked against GitHub’s current Copilot documentation, and checked again at the start of each engagement, because features and names change. The coding agent, for example, is now documented as the Copilot cloud agent.
The readiness exercise
Before the pilot starts, I review the Copilot setup for the pilot repository and its organisation:
- Policies and features. Which Copilot features and models are enabled at enterprise and organisation level, and whether that matches what the team will use.
- Content exclusion. Which paths are excluded, and where exclusion does and does not apply for the editors and modes your team uses. GitHub documents that support is uneven, so exclusion is treated as a helpful layer, not a secrets control.
- Repository rules. Branch protection or rulesets, required reviews and required status checks, so that no Copilot-assisted change reaches the main branch without CI and a human approval.
- Custom instructions. Whether the repository has a
.github/copilot-instructions.mdfile or path-specific instruction files, and whether they describe the real build, test and style rules. - Code review and cloud agent. Whether automatic Copilot review is configured, whether its reviews count towards approvals, and whether the cloud agent is enabled, for whom, and which of its workflow runs need approval.
The review-effort baseline
The baseline comes from your own pull-request data for the pilot repository, over a recent period: pull-request size, time to first review, review rounds, time to merge, reverts and escaped defects. We take the same measures during the pilot. Copilot’s own usage metrics are recorded alongside as the adoption measure, kept separate from the success measures.
The pilot workflow
The pilot team agrees a short set of rules and uses them on its normal backlog:
- Custom instructions are written by the team and changed through pull requests.
- Assisted pull requests stay under an agreed size; larger changes are split.
- Copilot code review runs as a first pass; a named human still approves.
- The cloud agent, if used, takes only agreed task types with good test coverage, works on its own branch, and its pull requests get an independent reviewer.
- Security-relevant code (authentication, payments, permissions) gets the same review it always did.
This follows the method I have published as Vibes Inside Guardrails: let developers move quickly inside constraints that CI and repository rules enforce, rather than relying on reviewers to notice everything.
The signature deliverable
You end with a repository-specific pilot, a review-effort baseline and a team usage playbook. Illustrative example of the comparison:
| Measure | Baseline | Pilot | Read with |
|---|---|---|---|
| Median pull-request size | measured | measured | Size rule introduced |
| Median time to first review | measured | measured | Copilot review as first pass |
| Review rounds per pull request | measured | measured | |
| Reverts and escaped defects | measured | measured | Same tracking both periods |
| Active Copilot users in team | dashboard | dashboard | Adoption, not success |
Illustrative example. Values come from your own repository and dashboard.
Acceptance, ownership and next steps
The developer experience lead accepts the pilot against the baseline and owns the playbook, the instructions and the measurement queries afterwards. If review load is the main issue across tools, see AI-assisted code review and release controls. For vendor-neutral agent controls, see coding-agent rollout. For Claude Code, see Claude Code enablement. The parent programme is AI engineering enablement for software teams.
Independent practitioner; not affiliated with, endorsed by or certified by GitHub or Microsoft.