Most teams' first experience with an AI coding assistant is the same. Open one repository, open one session, ask for something. It works well enough that you keep going — until the bug you're chasing lives in a different repo, and the assistant confidently fixes the wrong thing.
The platform is a marketplace: several dashboards, a couple of APIs, a shared auth service, roughly fifteen repositories over one database. I lead the frontend side of one product team, and this is what I measured going from a single-repo pilot to a workspace the whole team could clone.
Key Takeaways
- A test-writing pilot measured a 93.8% reduction in effort against my own baseline timings, with coverage moving from 68% to over 70%.
- One engineer, one task type, self-measured. The strongest trial available found experienced developers 19% slower with AI, on work close to the opposite of mine.
- Writing the conventions down is the work. A
CLAUDE.mdplus@testcasesannotations took output from plausible-but-foreign to roughly 80% shippable.- Single-repo sessions fail silently across services. They answer the wrong question confidently, because the cause sits in a repo they can't see.
- Fifteen repos under one workspace cut auth debugging from ~3 hours to ~30 minutes.
Starting small: a test-writing pilot
Test writing was the obvious first target. Repetitive, pattern-heavy, and nobody enjoyed it. The stack is React, Next.js and TypeScript, with Jest and React Testing Library.
Three choices made it work. First, a project-level CLAUDE.md carrying our test conventions: userEvent.setup() before render, getByRole over getByTestId, how we mock the API layer. None of that is original. It is close to Testing Library's own priority list, which is exactly the point — the conventions were already decided, just not anywhere the assistant could read them. Without the file, generated tests were plausible but didn't look like ours, which is worse than no tests, because a reviewer has to read every line to find out.
Second, @testcases blocks at the source, naming the scenarios that mattered.
/**
* @testcases
* - renders a skeleton while the quote list is loading
* - shows the empty state when the list resolves to zero items
* - surfaces a retry button when the request fails
* - calls onSelect with the quote id when a row is clicked
*/
export function QuoteList({ onSelect }: QuoteListProps) {
That is the difference between tests describing what a component does and tests describing what it is for. It also outlives the session: the next engineer to open the file reads the same list.
Third, one component at a time with a review loop. Output ran about 80% right, and the remaining 20% was editing rather than authoring, which is a different and much cheaper activity. Measured over the pilot, test-writing tasks came in at a 93.8% reduction in effort against my own baseline timings. Coverage moved from 68% to over 70%, and the tests were consistent because they all referenced the same file.
The number, and the study that disagrees
That is my stopwatch, on one task type. Putting it next to the research is more useful than defending it.
In July 2025 METR ran a randomised controlled trial: 16 experienced open-source developers, 246 real tasks, mature repositories they had worked in for around five years, each task randomly assigned AI-allowed or AI-disallowed. Developers were 19% slower with AI available. They had forecast a 24% speedup beforehand, and afterwards still reported a 20% speedup.
That last part is the one to sit with: on the same tasks, by the same people, the measurement and the feeling disagreed by nearly forty points.
It doesn't invalidate my number, and the authors say so — they list "AI systems do not currently speed up many or most software developers" among the conclusions their study doesn't support. What it does is locate it. METR measured experts doing judgement-heavy work in code they knew intimately. I measured scaffolding, environment setup and cross-service archaeology, where the bottleneck is retrieval rather than judgement.
Stack Overflow's 2025 survey is the harder one. Forty-five percent of developers say debugging AI-generated code takes them more time, and the top frustration, at 66%, is output that is almost right but not quite. The numbers further down this post say the opposite. The difference, I think, is that I'm not asking an assistant to debug from a blank slate.
Where it broke: one repo, one session
A login failure in a dashboard. The assistant, opened in that repo, sees the request that failed and the component that made it. It can't see the auth service that issued the token, or the shared schema underneath. So it proposes retry logic and a defensive null check, for a problem whose cause is a JWT secret that drifted between two services.
The failure mode isn't that the assistant was wrong. It was confident and plausible inside the only boundary it could see. A wrong answer you can spot costs a minute. A reasonable answer to a question you didn't ask costs the afternoon.
The knowledge that would have ended that session existed. It lived in people's heads and old Slack threads: which secrets must align across services, that the auth service masks OAuth errors as bare 500s, how central migrations are meant to run. Every engineer rediscovered it from scratch, with or without an assistant.
An AI-ready workspace
The fix is deliberately boring. Markdown and git, no new tooling.
workspace/
├── CLAUDE.md # platform map: what each repo is, how they talk, what's shared
├── skills/ # reusable procedures: "rebuild auth locally", "trace a token"
├── auth-service/
│ └── CLAUDE.md # repo-specific conventions and gotchas
├── core-api/
│ └── CLAUDE.md
└── ...
All repos are cloned side by side so that a session opened at the root reads across service boundaries, and a workspace-level CLAUDE.md maps which repo owns what and how auth flows between them. Per-repo files carry local conventions, and a shared skills folder holds the recurring procedures. Nothing here is clever. It is all text in git, which means it gets reviewed, it gets blamed, and it outlives whichever assistant is current.
I used it daily across frontend and backend work for four months before proposing it to anyone.
That fourth row is what made me write the proposal up. I went looking for a published benchmark and couldn't find an honest one. The nearest real research, Google's study of onboarding and ramp-up, measures ramp toward median contribution over months rather than hours to a working laptop. Take the number as one engineer's observation. What changed wasn't the assistant. Somebody finally wrote the setup down in a form that could be executed rather than read.
What I would tell another tech lead
- Documentation is the product. An assistant multiplies whatever context exists. Zero context, zero multiplier. The same holds once you let an agent run in a loop: it multiplies whatever check exists.
- Start with boring, pattern-heavy work. That is where the evidence is kindest, and where judgement isn't the bottleneck.
- Time your own baseline first. The 93.8% means something only because I measured the before.
- Cross-repo is where the pain is, and where a single-repo setup fails quietly rather than loudly.
- Keep it tool-agnostic. Markdown and git outlast any particular assistant.
- Present your numbers as what they are. Directional single-engineer measurements are useful. Dressing them up as studies is how you lose the room.
The leverage was never in the model. Every team has roughly the same models. It is in how much of your operational knowledge is written down where an assistant can read it, which is, inconveniently, the same place a new engineer would have looked.