Edison SinaniAI implementation architectBook a call

AI Team Playbook / Rule 11 of 12

Measure five numbers from day one

The team records five numbers every week from the tracker and the git host, starting before the playbook changes anything, and shows a sponsor nothing else.

September 16, 2026 · About 2 minutes

Contents

The playbook

  1. Overview

Before code

  1. 1Write the constitution first
  2. 2One instruction file, under 200 lines
  3. 3Prose is advice, hooks are enforcement
  4. 4Three gates before any code
  5. 5A walkable demo before any screen

Building

  1. 6One task, one worktree, one pull request
  2. 7Every contract has a file and two tests

Reviewing

  1. 8AI review first, and filtered
  2. 9Human review by risk tier
  3. 10Done means the evidence is in the pull request

Over time

  1. 11Measure five numbers from day one
  2. 12Protect the people who are still learning

Appendix

  1. AInstall checklist
  2. BOther tools, limits and sources
  3. CTightening the process
  4. DRun the agent with no entitlements

Get the kit

Who owns it
The lead records them and reads them at the Friday checkpoint. Cost per developer is seen by the lead only.
In the kit
docs/measurement.mddocs/governance.mddocs/roles-and-rituals.md
Enforced by
Five minutes on the agenda of the weekly checkpoint. No number goes in a status report that did not come from this file.

Why

Experienced developers in the best randomized study so far were 19 percent slower with AI while believing they were 20 percent faster [5]. Without a baseline, the only evidence in six months will be how people feel, and how people feel is wrong. The 2025 DORA report, from nearly 5,000 respondents, found that higher AI adoption went with higher delivery throughput and higher delivery instability at the same time [15]. The team’s job is to be the exception, and this is how it will know.

How

Five numbers, one row per week, from tools the team already has.

Number From Why it matters
Pull requests merged The git host The volume that goes up first
Median pull request size, in changed lines, excluding generated files The git host The thing that grows silently
Median time from opening a pull request to merging it The git host The review queue, where the cost lands
Change failure rate: pull requests needing a fix or revert within 7 days Tracker labels hotfix and revert

Quality. Bugs per developer rose 54 percent in the Faros data [1]
Rework: pull requests with more than two review rounds The git host The signal that reviewers are tiring

Do not record lines of code written, the acceptance rate of suggestions, or hours saved from people’s own estimates. None of these predicted delivery outcomes in any published study, and the first two reward the wrong thing.

Record cost privately.

Anthropic reports about 13 dollars per developer per active day on average, 150 to 250 dollars a month, and under 30 dollars a day for 90 percent of users [14]. The per-user figure is for the lead to spot a session that looped, not for ranking people. Say that out loud when the team starts. Most overruns are a session that looped, and the fix is in the task, not the person.

What good looks like after one milestone. Pull requests merged goes up. Median size stays flat or falls, because rule 6 holds. Time to merge stays flat, because rules 8 and 9 absorbed the volume. Change failure rate stays flat. If the first number is up and the other three are flat, the setup is working. If time to merge doubled, tune the automated reviewer before adding people.

What to show a sponsor. The five weekly numbers as a chart, with the date the playbook was installed marked on it. Nothing else. Any company’s published headline percentage came from a setup that is not this team’s.

The rituals that read the numbers. A Friday checkpoint of twenty minutes with four questions: can we demo what we built, is there a blocker someone is avoiding, are we building something not on the plan, what is the one thing for next week. Then five minutes on the numbers. And a milestone retrospective that retires any rule nobody needed. The playbook is meant to shrink over time, not grow.

References

  1. 1Faros AI, "The AI Engineering Report 2026: The Acceleration Whiplash", April 2026. Telemetry from 22,000 developers and more than 4,000 teams over two years. www.faros.ai/blog/ai-acceleration-whiplash-takeaways
  2. 5METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", July 2025. Randomized controlled trial, 16 developers, 246 tasks. metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
  3. 14Anthropic, Claude Code documentation, "Manage costs effectively". code.claude.com/docs/en/costs
  4. 15Google Cloud DORA, "State of AI-assisted Software Development", 2025. Survey of nearly 5,000 technology professionals. dora.dev/research/2025/dora-report/