Edison SinaniAI implementation architectBook a call

AI Team Playbook / Rule 11 of 12

Measure five numbers, baseline first

The team records five numbers every week from the tracker and the git host, backfilled for the 8 to 12 weeks before the playbook is installed, and shows a sponsor only these five.

September 16, 2026 · About 3 minutes

Contents

The playbook

  1. Overview

Before code

  1. 1Write the constitution first
  2. 2One instruction file, under 200 lines
  3. 3Enforce the rules with settings and hooks
  4. 4Three gates before any code
  5. 5A walkable demo before any screen

Building

  1. 6One task, one worktree, one pull request
  2. 7Every contract has a file and two tests

Reviewing

  1. 8AI review first, and filtered
  2. 9Human review by risk tier
  3. 10Done means the evidence is in the pull request

Over time

  1. 11Measure five numbers, baseline first
  2. 12Protect the people who are still learning

Appendix

  1. AInstall checklist
  2. BOther tools, limits and sources
  3. CTightening the process
  4. DRun the agent with no entitlements

Get the kit

Who owns it
The lead records them and reads them at the Friday checkpoint. Cost per developer is seen by the lead only.
In the kit
docs/measurement.mddocs/measurement-log.csvdocs/governance.mddocs/roles-and-rituals.md
Enforced by
Five minutes on the agenda of the weekly checkpoint. No number goes in a status report that did not come from this file.

Why

In a 2025 randomized study by METR, experienced developers were 19 percent slower with AI while believing they were 20 percent faster [5]. Without a baseline, the only evidence in six months will be how people feel, and that is a poor measure. The 2025 DORA report, from nearly 5,000 respondents, found that higher AI adoption went with higher delivery throughput and higher delivery instability at the same time [15]. The five numbers below show which way this team is going.

How

Backfill the baseline first. Most of it already exists. Pull request counts, sizes, merge times and review rounds for the last 8 to 12 weeks are in the git host’s history. Pull them on day one, before the kit is installed, and the baseline is there on the first day.

Then five numbers, one row per week in docs/measurement-log.csv, from tools the team already has.

Number From Why it matters
Pull requests merged The git host The volume that goes up first
Median pull request size, in changed lines, excluding generated files The git host

It grew 154 percent on teams with high AI adoption in the Faros data [2]
Median time from opening a pull request to merging it The git host How long changes wait for review
Change failure rate: pull requests needing a fix or revert within 7 days Tracker labels hotfix and revert

Quality. Bugs per developer rose 54 percent in the Faros data [1]
Rework: pull requests with more than two review rounds The git host How often a change needs repeated review

Do not record lines of code written, the acceptance rate of suggestions, or hours saved from people’s own estimates. I know of no published study in which any of these predicted delivery outcomes.

Label agent-assisted work. The kit’s CI adds the claude-code-assisted label to every pull request whose template says Tool: Claude Code, so any of the five numbers can be split by it.

Record cost privately.

Anthropic reports about 13 dollars per developer per active day on average, 150 to 250 dollars a month, and under 30 dollars a day for 90 percent of users [14]. The per-user figure helps the lead spot a session that looped. It is never used to rank people. Say that out loud when the team starts. Most overruns come from a looping session, and the fix is in the task.

What good looks like after one milestone. Pull requests merged goes up. Median size stays flat or falls, because rule 6 holds. Time to merge stays flat, because rules 8 and 9 handle the extra volume. Change failure rate stays flat. If the first number is up and the other three are flat, the setup is working. If time to merge doubled, tune the automated reviewer before adding people.

What to show a sponsor. The five weekly numbers as a chart, with the date the playbook was installed marked on it. Leave out other companies’ published percentages. They came from different setups.

The weekly and milestone reviews. A Friday checkpoint of twenty minutes with four questions: can we demo what we built, is there a blocker someone is avoiding, are we building something not on the plan, what is the one thing for next week. Then five minutes on the numbers. And a milestone retrospective that retires any rule the team did not use. The playbook is meant to shrink over time.

References

  1. 1Faros AI, "The AI Engineering Report 2026: The Acceleration Whiplash", April 2026. Telemetry from 22,000 developers and more than 4,000 teams over two years. www.faros.ai/blog/ai-acceleration-whiplash-takeaways
  2. 2Faros AI, "The AI Productivity Paradox", July 2025. Telemetry from more than 10,000 developers across 1,255 teams. www.faros.ai/blog/ai-software-engineering
  3. 5METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", July 2025. Randomized controlled trial, 16 developers, 246 tasks. metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
  4. 14Anthropic, Claude Code documentation, "Manage costs effectively". code.claude.com/docs/en/costs
  5. 15Google Cloud DORA, "State of AI-assisted Software Development", 2025. Survey of nearly 5,000 technology professionals. dora.dev/research/2025/dora-report/