DEV Community

Cover image for The Review Tax: Why 81% of Developers Are Buried in AI Code Review
Harsh
Harsh

Posted on

The Review Tax: Why 81% of Developers Are Buried in AI Code Review

Verifying code took the speed savings

Just give it to AI might be the most dangerous phrase in software development right now.

I've said it myself. Handed off a task, watched clean-looking code come back in seconds, skimmed it, and moved on because it looked right and the tests were green.

Then I reviewed a PR that wasn't mine to write, just mine to check. AI-generated, clean, organized, passing every test I threw at it. I approved it the way I'd approve anything that looked competent on the surface.

The bug showed up later. Not in review, not in testing. In production, after the code had already been trusted for a while. Nothing about it had looked wrong. That was the actual problem: it wasn't obviously wrong, it was quietly wrong, in the specific way that only announces itself once real conditions hit it.

I went back afterward and sat with that PR properly. Not skimming this time. Actually reading it, actually understanding what it was doing and why, actually treating the review like the real work instead of the formality before merging. It took a lot longer than approving it had. It's the only way I'd have caught it before production did.

Since then, I don't rush AI-code reviews anymore. I give them the time writing the code apparently didn't need. And it turns out I'm far from the only one who's landed there. 🧵


The Number That Explains What I Was Feeling

According to Harness's 2026 State of Engineering Excellence Report, a survey of 700 engineering practitioners across the US, UK, India, France, and Germany, 81% of developers now spend more time in code review since their teams adopted AI tools. 28% report review time increasing by 30% or more.

Here's the trade nobody advertised clearly: AI tools cut time-to-PR by roughly 58%. But those same PRs then sit in review 4.6x longer than before. Review time per developer is up an estimated 11.4 hours a week.

The speed didn't disappear. It moved. It went from "time spent writing" to "time spent verifying," and verifying turns out to be the harder, slower half of the job.

I've started calling this the Review Tax, and I don't think that's an exaggeration. Nearly 31% of developer time is now going toward what researchers are calling "invisible work": reviewing AI output, fixing the subtle bugs it introduced, and context-switching to explain code nobody on the team originally wrote by hand.


Why Reviewing AI Code Is Genuinely Harder Than Reviewing Human Code

This isn't just "more code to review." It's a fundamentally different, more draining kind of reviewing.

When you review a colleague's code, you're checking work from someone with a track record. You know their habits, their typical mistakes, roughly how they think. When you review AI-generated code, none of that context exists. You're evaluating output from something that writes with total, uniform confidence whether it's right or wrong, and gives you no signal to tell the two apart.

Developers surveyed named the friction directly: reviewing AI code for accuracy (53%), fixing subtle bugs in AI-generated code (52%), explaining AI-generated code to teammates (48%).

Researchers analyzing this trend describe a widening gap between two kinds of developers on a team. Builders: people who genuinely understand the system, make deliberate choices, and take ownership of what they ship. Coasters: people who generate code, open large PRs, and move on without building any real understanding of what they just shipped.

The uncomfortable part is that AI makes Coasters look more productive. They open more PRs. They touch more lines. A manager tracking output alone sees a high performer. The Builders reviewing that same work know they're the ones actually catching what would otherwise break in production, quietly, unglamorously, off any dashboard that measures velocity.


The Accountability Problem Nobody Wants to Say Out Loud

The Godot Foundation recently banned AI-authored code contributions entirely. Their reasoning, in their own words: "AI cannot take responsibility, and we can't trust heavy users of AI to understand their code enough to fix it."

One maintainer described the deeper cost: when your review feedback is being absorbed by a process instead of mentoring an actual future contributor, it becomes much harder to justify spending your own time on review at all.

That's the part of the Review Tax that doesn't show up in any productivity metric. It's not just slower. It's less motivating. Reviewing a human's code, even when it needs work, is partly an investment in a person who'll get better because of your feedback. Reviewing AI output that will forget every correction the moment the session ends removes that entire dimension. You're not mentoring. You're just, endlessly, catching things.


What Actually Helps

I don't think the answer is reviewing less carefully, and I don't think it's avoiding AI either. Here's what's actually shifted for me and what the research backs up.

1. Catch issues earlier, not just more carefully at the end.

The cost of a bug climbs sharply the later it's caught: roughly 1x if it's found in design, about 6x during coding, and anywhere from tens to over a hundred times that if it reaches production. Pre-commit checks catch a large share of issues before they ever reach a human reviewer's queue, cheaply.

2. Stop measuring volume.

Lines of code and PR count were always shaky metrics for productivity. With AI in the loop, they're actively misleading, they reward the exact behavior driving the Review Tax up.

3. Make authors explain their own PRs.

If the person opening the PR can't explain their own approach in a couple of minutes, it doesn't merge yet, regardless of how clean the diff looks. This single rule filters out most of what turns into a slow, painful review later.

4. Actually track the invisible work.

Only about 38% of organizations currently track time spent reviewing AI-generated code specifically. Most developers, 94% in one survey, say tech debt, validation time, and burnout aren't showing up in the metrics leadership actually looks at. You can't fix a cost nobody's measuring.

5. Say the norm out loud, don't just assume it.

"We treat AI as a drafting tool, not a shipping tool" sounds obvious until you watch a team operate without ever agreeing to it explicitly. Teams that state this clearly review differently than teams where everyone's silently guessing at how much scrutiny is expected.


The Honest Version

The Review Tax isn't only a productivity problem. It's a people problem wearing a productivity problem's clothes.

When your most careful reviewers start going quiet on PRs, that's rarely them relaxing. More often, it's the early, silent shape of burnout, the kind that doesn't announce itself until someone's already decided to leave.

Teams that build real norms around AI-assisted review will keep the developers who actually understand their systems. Teams that don't will eventually be staffed by people shipping a lot of code that nobody, including the person who opened the PR, could fully explain if asked.

The question worth sitting with isn't "should we use AI to write code." It's "who's actually accountable when nobody on the team wrote the code they're shipping."


Have you felt the Review Tax on your own team? What's the worst AI-generated PR you've had to untangle, and did your review process change after it? I'll go first in the comments. 👇

Top comments (40)

Collapse
 
gramli profile image
Daniel Balcarek

To be honest, we still use AI mostly as an assistant. It generates relatively small parts of the code, so this “AI review tax” doesn’t affect us that much yet.

What also caught my eye was the survey size: 700 developers across five countries. I’m not sure how representative that sample really is, especially for Europe.

From my experience, even though developers use AI daily, not every company has fully integrated it into the development process or approved the budget and tooling to the point where developers mostly generate code and then spend their time reviewing it. At least not yet. 😀

Collapse
 
harsh2644 profile image
Harsh

Daniel that's a fair point on the sample size and I should've been more careful flagging it in the article. 700 across five countries is a reasonable survey, but it's still one survey, and generalizing 81% of developers from it probably overstates how universal this is right now. representativeness for any single country in that mix, Europe included, is a real limitation worth naming rather than glossing over.

and your point about adoption stage matters more than I gave it credit for. the Review Tax as I described it assumes AI is generating a large enough share of the codebase that review becomes the bottleneck. if AI's still mostly assisting on small pieces, the dynamic looks completely different, you're reviewing normally-sized changes with some AI-assisted sections inside them, not reviewing large AI-generated diffs as the default unit of work.

probably the more honest framing is that this is what happens at a certain point on the adoption curve, not a universal state every team using AI is already in. worth being clearer about that distinction.

Collapse
 
yielab profile image
Santiago yie

That's totally true! Thanks for the tips. I'm wondering: what are the best practices when writing a prompt to make human review easier and set proper constraints? For example, how do you phrase things so the AI avoids over-engineering, checks existing file structures to follow the chosen architecture, or replicates the pattern of an existing code for a new feature or bug fix?

Collapse
 
harsh2644 profile image
Harsh • Edited

Thanks Santiago few things that have actually helped, not just theory: for avoiding over-engineering be explicit that you want the simplest solution that solves the stated problem, nothing more. something like use the standard library or an existing pattern in this codebase before adding anything new and default to the shortest solution that passes the requirement. I wrote about a tool called Ponytail a while back that basically formalizes this into a checklist the agent runs through before writing anything, worth looking into if you want it enforced rather than just requested.

for following existing architecture: don't describe the pattern, point at it. follow the same structure as [specific existing file/function] works far better than explaining the pattern in prose, because the agent is matching against a concrete example instead of interpreting your description of one. vague architectural instructions get vague architectural compliance.

for replicating patterns on new features: same principle, give it the actual sibling implementation as context, not a summary of what that implementation does. this new endpoint should mirror the error handling, naming, and structure of [existing endpoint], changing only what's different about this specific case produces far more consistent output than describing your conventions abstractly.

the underlying thing all three have in common: specificity and concrete examples beat well-written prose instructions almost every time. the agent isn't reasoning about your intent as much as pattern-matching against whatever you actually gave it to match against.

Collapse
 
yielab profile image
Santiago yie

Thanks for the detailed breakdown, Harsh! This makes total sense—treating the AI as a pattern-matcher rather than a reasoner changes how I'll approach these. I'm going to look into Ponytail, too

Collapse
 
effessdev profile image
EffessDev

Here is what I generally do:

  • Ask it to tell me what it is going to do. That enables me to find problems earlier.
  • Split the problem into smaller steps so it's easier for the AI to generate and us to review, and do it one after the other (commit each step separately instead of a single big commit).
  • Ask if there is something it needs to know to complete the task, if something is unclear, and what assumptions it made.
  • Tell it what NOT to do, and what our priorities are (e.g., "Do not create new api routes, reuse the existing ones. Use a simple UI for now, we will upgrade it in a later commit.").

I think current AI agents already heavily check the existing file structure. I add custom patterns and rules in AGENTS.md.

Note: I am a beginner so please point out if there are mistakes.

Collapse
 
yielab profile image
Santiago yie

Thanks. Asking the AI to state its plan and assumptions before coding, while using an AGENTS.md file for custom patterns, is a helpful strategy.

Collapse
 
unitbuilds profile image
UnitBuilds

Imo, this is exactly why AI is a senior dev's tool. Not a junior and while code generation is a 10-100x speedup, it should be a flat 2x productivity, because the time you save writing the code, you should spend validating it. The other note, is that we should remember, subtle errors in production have existed since before AI, it's not an AI problem, it's a business operations problem. If a senior dev uses AI, reviews the outputs, corrects it as they go, validates when they're done, they still produce PRs at 2-3x the rate, at the same quality as if they had written it all by hand, because they spend more time validating, than writing.

To rely on writing it as a quality check, is fundamentally flawed, if you ever relied on that as a metric of correctness, then you survived on sheer dumb luck till now. People make the same mistakes AI does, except AI writes 100x more code, with more safety checks, tests, etc. That doesnt make a human more right, or less wrong, it's just a fundamental problem discovered later down the development pipeline. If your calculation logic is off, if you just wrote it, no tests, no gates, nothing, you've made the exact same mistake the AI did, with all green tests and gates. That is why it's less important to look at how it validates and more important to look what it's validating against. if you typed 55x and the AI implemented 55x, but it was meant to be 5.5x, the error is the same, except the AI built the infrastructure too to prove it's 55x. That's on the developer, not the AI, because they're meant to view the logic as they develop it. That's an error that would slip through QA testing and reviewing, until production suddenly has incorrect values. The bug is the same, but you focus less on what's important than you should. If you're validating the AI's tests, catch blocks, etc. Before you validated it's formula, you wasted 99% of the time it took to find the bug.

Personally though, I dont think relying on AI to generate code is a sound strategy for 99% of use-cases. Rather use something like Roslyn to generate template structures and have the AI fill in the blanks. That way you keep context clean, you reduce review necessary work and your codebase is cleaner and more maintainable, all at once. Eg. V.A.L.I.D. uses Roslyn to generate a MCP and Unit Tests, driven by how you define the ValidObjects in the DTO and how you set up the HTML markup, Roslyn generates everything inbetween, so all AI/You have to do, is design the UI layout and define the properties they represent. 2 files to view, instead of 20. That's how you use AI effectively, by constraining what it needs to do, to what a human reviewer can easily validate and let the system test it's logic, not have the AI write it's own tests.

Collapse
 
harsh2644 profile image
Harsh

Flat 2x, not 10-100x" is the reframe that should replace every AI productivity claim I've seen marketed. the 10-100x number only holds if you count generation speed and pretend validation is free. the moment validation gets its fair share of the timeline, the real number is exactly what you're describing, faster than writing by hand, nowhere close to the headline multiplier.

the 5.5x vs 55x example is the clearest illustration I've read of the actual failure mode. the bug is identical whether a human typed 55 or AI generated it, the difference is that AI also builds the tests, the gates, the green checkmarks around the wrong number, which makes the error look more validated than a human's unverified 55 ever would have looked. that's a genuinely dangerous property: confidently wrong at scale, with infrastructure vouching for the wrong answer.

it's less important to look at how it validates and more important to look at what it's validating against" might be the single most useful sentence in this whole comment section. reviewing test coverage and catch blocks before confirming the underlying formula is correct is exactly backwards, and I think a lot of review processes, mine included, default to that order because tests are easy to check and business logic correctness is hard to check.

the Roslyn-plus-DTO approach you're describing (constrain what needs generating down to what's actually reviewable, let the system own everything else) is the same principle from your earlier comment, applied to the review-tax problem specifically: you don't reduce the review tax by reviewing faster, you reduce it by shrinking what genuinely needs review down to the two things a human is positioned to catch, and removing the rest of the burden structurally rather than trying to read it more carefully.

Collapse
 
unitbuilds profile image
UnitBuilds

Exactly, that's kind of why I built V.A.L.I.D. the way I did, because if you ever tried letting AI generate anything using CSLA, you'd know it's garbage as managing Business Objects, Data Access Layers, Endpoints, UI Bindings and horribly inconsistent with SQL Repo connection handling. So take out all the things it can make a mistake with, beyond what's genuinely up for interpretation (UI design and DTO) and generate the rest using fixed structures for consistency, accuracy and constraint. The amount of times i had to catch AI for putting AllowAnonymous() in a fast endpoint configuration... That's probably 1 of the most severe network security nono's you can do, yet it's default behavior for alot of models... You can try and fix it with skills files, but that just 'suggests' a better method, rather than strictly enforcing it the way a Roslyn generated template is.

Thread Thread
 
harsh2644 profile image
Harsh

AllowAnonymous() defaulting to open across models is the kind of thing review carefully was never going to catch, it's a reflex baked into training, not a one-off slip.

that's the throughline for this whole thread actually: interpretive stuff (UI, DTO shape, business rules) is where AI adds real value. anything with one non-negotiable correct answer (auth defaults, data access, connection handling) is where "generate it and review carefully" stops working, no matter how careful the review is.

a skills file suggests a better pattern. a Roslyn template removes the bad one from what's even possible to generate. those aren't the same fix at different strengths, they're different categories.

Collapse
 
edmundsparrow profile image
Ekong Ikpe

Your introduction has most of what I would have loved to contribute nevertheless. That part to me is not as emphasized as I think it ought to be.
Even if a senior developer uses an AI for a single line of code without knowing the details or surrounding implications the end result will be babash 🤣 : Without the AI software is not perfect right from when I learnt how to use a Windows PC in the earliest 2000, I saw bug updates, patches, etc and till now updates are there to fix and improve code, UX, UI, the software entirely.

With or without AI usage output will be poor if poorly inputted.
The wrong Assumptions is the "intelligence" in the AI and ignoring the artificial in it. I hope I make small sense 🤦

Collapse
 
unitbuilds profile image
UnitBuilds

Exactly, it's not to do with AI makes mistakes, it's mistakes happen, that's why you have to validate and that goes for anyone and everyone, be it a monkey an AI or a human, you validate, else it might as well have not existed in the first place, because you dont know what it does or that it does it correctly.

Collapse
 
edmundsparrow profile image
Ekong Ikpe

The bias in faulting the AI for people debugging longer than personalizing the syntax responsibility is what attracts me to most threads.
Look at my case for example: The thing is that I have written more code my entire life in the shortest time I began to use AI assistants ~3 years ago than the last 20+ years.. 1st thing I went after the 1st time I encountered a chatbot that seemed more knowledgeable than my regular Google-Search use case was - "can you code"? Now I'm enjoying debugging the code and learning faster. Imagine there was no chatbot or AI 🤷 would I ever build a Todo app?

The thing is how we see, use, interact, romance or whatever with the AI it reflects back our intelligence.
"Using a keyboard doesn't make me smarter than someone using a pen 😂.

Collapse
 
leob profile image
leob

"The speed didn't disappear. It moved. It went from "time spent writing" to "time spent verifying," and verifying turns out to be the harder, slower half of the job." - aren't we then even getting any productivity gains at the end of the day? And at what cost - a lot of people like writing code more than reading it, and the risk is we lose some of the 'skills of the trade' ...

For these reasons I'm arguing for an approach where we still write some of the code ourselves, manually: the more interesting, logic heavy code, while we let AI write the less critical and more run-of-the-mill code (CRUD, boilerplate) ...

Collapse
 
harsh2644 profile image
Harsh

Thanks Leob Honest answer on the first question: net productivity gain, yes, mostly, but smaller than advertised. 58% faster to PR minus 4.6x longer in review still lands ahead of zero, just nowhere near the multiplier most pitches lead with. the real cost isn't that the gain disappears, it's that it gets quietly smaller every time review gets skipped or rushed, which is exactly how you end up with a bug in production instead of a bug in review.

the people like writing more than reading point deserves more attention than it usually gets. it's not just preference, it's how the skill gets built and maintained in the first place. you don't develop debugging instinct or a feel for a system's edges by reading someone else's output, you develop it by getting stuck and working through it yourself. reviewing doesn't build the same muscle writing does, even when you're reviewing carefully.

your split (write the interesting, logic-heavy code yourself, let AI handle CRUD and boilerplate) matches what's come up a few times in this comment section from different angles: constrain AI to the parts that are genuinely low-stakes or mechanical, keep the parts with real judgment calls in human hands. that also happens to be the split that keeps the skill from atrophying, because the code you'd actually want practice on is the code you're still writing.

Collapse
 
leob profile image
leob • Edited

Agreeing 100% with everything you said, we're on the same wavelength! I think that's a responsible way to use AI for coding, with more advantages than disadvantages (also long-term) ...

If, on the contrary, we just hand off everything to AI, with insufficient checking and reviewing, that's simply creating a ticking time bomb ...

Thread Thread
 
harsh2644 profile image
Harsh

Appreciate that, Leob Ticking time bomb is exactly the right word it's not about the gain disappearing, it's about it quietly shrinking until one day it's just not there anymore.

Glad we're on the same wavelength on this one. 🙏

Collapse
 
effessdev profile image
EffessDev • Edited

If we carefully verify the AI output before finishing PRs, we will get fired for being slow. When we quickly finish PRs without verifying the AI output, we get fired for creating a major bug in production 🤦

But I think in some areas it's ok to just let the AI do whatever it needs and skip the review step. In critical areas, I still write code by hand because verifying the AI-generated code in detail actually takes me longer than writing it manually. In those cases, I find AI autocomplete really useful.

Today, while I was testing Qoder, I asked it to integrate a payment gateway into my Next.js website to allow people to buy credits. When I looked at the code in the api route, the credits are directly incremented after verifying the payment. I asked "Does triggering the api route after one initial payment again and again allow people to increment credits indefinitely?"

It said "yes" 😭😭😭

Collapse
 
harsh2644 profile image
Harsh

The damned-if-you-do dynamic you opened with is real, and I don't think there's a clean fix for it beyond someone senior explicitly protecting review time as a metric, rather than leaving it to individual devs to justify being slow.

but the payment gateway story is the one that's going to stick with me. asking the AI does calling this endpoint repeatedly let someone increment credits indefinitely and getting a straight yes back is somehow both the funniest and most damning thing in this entire comment section. the AI didn't just fail to catch it while writing the code, it could articulate the exact vulnerability the moment someone asked the right question, which means the knowledge to prevent it was in there, it just never got triggered until you specifically interrogated the output instead of trusting it.

that's actually a strong argument for what you said earlier in the comment, non-critical paths can probably tolerate skipping the deep review, but did I just verify a payment endpoint is exactly the kind of area where writing it by hand, or at minimum interrogating it like you did here, isn't optional. glad you caught that one before Qoder's answer became a very expensive lesson instead of a very funny comment. 😭

Collapse
 
effessdev profile image
EffessDev

Luckily I was testing it on my personal project. Imagine I was working on someone else's 💀

Collapse
 
capestart profile image
CapeStart

I don't mind reviewing AI-assisted code. I mind reviewing 800-line PRs that could've been four smaller ones.

Collapse
 
harsh2644 profile image
Harsh

That's a sharper distinction than AI review is harder and I think it's the more honest one. an 800-line PR was always going to be exhausting to review, human-written or not. AI just makes it easier to accidentally generate 800 lines in one sitting, because nothing about the tool naturally stops at a reviewable size the way manually typing four separate features tends to.

the fix for this one doesn't even need new tooling, just enforcing the same PR-size discipline good teams had before AI, and treating this should've been four PRs as a valid review comment on its own, before anyone even reads the logic.

Collapse
 
tech_grundy profile image
The Tech Grundy

"AI makes Coasters look more productive on paper, while Builders absorb the Review Tax off any dashboard" is an incredibly sharp observation.

The rule about making authors explain their own PRs in a couple of minutes before merging is absolute gold. It forces ownership back onto the person opening the diff and stops people from using AI as a shortcut to skip learning the codebase. Outstanding writeup, Harsh!

The point about the missing mentorship element hit home.

When you review a junior developer's code, leaving detailed feedback is an investment in a person who will grow and improve next time. When you spend that same energy catching subtle bugs in AI-generated code that will forget the feedback by the next session, it turns code review into a purely exhausting tax. Spot-on analysis of what's driving reviewer burnout!

That stat about speed moving from "time spent writing" to "time spent verifying" explains so much of the current dev fatigue!

AI code is terrifyingly good at looking right while being quietly wrong under production edge cases. Calling out the shift to "invisible work" and treating AI as a drafting tool rather than a shipping tool is spot on. Great perspective!

Collapse
 
harsh2644 profile image
Harsh

Thank you so much, genuinely 🙏

the Coasters/Builders line resonating is exactly the reaction I was hoping for, because it's the part of this that's easiest to miss when you're only looking at output metrics. a manager tracking PR count has no way to see who's actually absorbing the tax.

and you've articulated the mentorship point better than I did in the article itself. energy that used to build a future maintainer now just evaporates into catching bugs a session will forget is the exact shape of why this specific kind of review burnout feels different from ordinary busy-ness. it's not just more work, it's work that stops accumulating into anything.

really appreciate you reading this closely enough to pull out all four threads separately. means a lot.

Collapse
 
glenallen profile image
Glen Allen

The bigger opportunity may be to reduce what actually reaches human review rather than trying to make humans review AI output faster. If deterministic checks, architectural constraints, and smaller changes can eliminate predictable classes of mistakes before a PR is opened, reviewers can spend their time on business logic and decisions that genuinely require human judgment. That seems like a more sustainable way to address the review tax.

Collapse
 
harsh2644 profile image
Harsh

Reduce what actually reaches human review is a better target than make review faster and I think the article leaned too hard toward the second one without saying so directly. faster review is still reviewing everything. this is a different strategy entirely: shrink the set of things that need a human's judgment in the first place.

the framing that makes this click is separating predictable mistakes from decisions that genuinely require judgment. deterministic checks and architectural constraints can catch the first category before a PR even opens, the kind of error that's the same shape every time and doesn't actually need a human brain to catch it. what's left for review is business logic and the tradeoffs a system can't evaluate on its own, which is exactly where a reviewer's time is worth spending.

this is the same shift a few other comments on this thread have landed on from different angles one described using Roslyn-generated templates so AI only fills in narrow, well-typed slots, someone else talked about capability scoping instead of trusting the model's judgment. all three are the same underlying move: stop trying to make humans better at catching everything, and shrink what needs catching down to what actually requires a human.

Collapse
 
suraj09 profile image
Suraj Suradkar

I keep wondering if part of the “Review Tax” is actually a context problem, not just a code-generation problem.

A reviewer isn't only asking “does this code work?” They're also asking “why is the system designed this way, what constraints already exist, what decisions led here, and what might this change break somewhere else?”

An AI can be given the relevant files, an AGENTS.md, and even an example implementation, but that's still very different from having the project's accumulated decisions and context available consistently.

Maybe that's why generating a small diff can still create a surprisingly large review burden: the reviewer has to reconstruct the context that the agent didn't have.

I'm curious whether you think the long-term solution is mainly better review workflows, or whether coding agents eventually need a persistent understanding of the project itself to reduce that burden.

Collapse
 
harsh2644 profile image
Harsh

Thanks Suraj this reframes the whole article in a way I think is more accurate than what I actually wrote. I was treating the Review Tax as AI writes code faster than humans can check it. you're pointing at something underneath that: the reviewer isn't just checking correctness, they're reconstructing context the agent never had access to in the first place, and that reconstruction work is invisible in every metric that just measures diff size.

that also explains something that confused me while writing this: why a small, clean diff can still take forever to review. it's not the size of the change, it's the size of the gap between what the agent knew when it wrote it and what the reviewer needs to know to evaluate it safely.

honest answer to your actual question: I don't think it's either or, but if I had to bet on which one matters more long term, it's persistent project understanding, and here's why. better review workflows are a way of managing the symptom, they make reconstructing context faster or more structured, but a human is still doing the reconstructing every time. an agent with genuine persistent understanding of the project's accumulated decisions would mean the reviewer isn't rebuilding that context from scratch on every diff, because the thing that generated the code already had it.

that said, I'd bet against this arriving cleanly anytime soon. even AGENTS.md and example implementations are still a snapshot handed to the agent per session not something the agent actually retains and updates its own understanding of over time. until that gap closes review workflows built explicitly to surface why was this decision made alongside the diff are probably the realistic stopgap, not because they solve the problem, but because they're the thing actually available right now.

Collapse
 
suraj09 profile image
Suraj Suradkar

Yeah, I think that distinction between the “context the agent had” and the “context the reviewer needs” is probably the most interesting part of this.

What I'm wondering now is whether persistent project understanding would need to be more than just storing past decisions.

For example, if a decision changes six months later, the system would need to know that the old context is no longer authoritative, what replaced it, and which parts of the code or other decisions were affected.

Otherwise, persistent memory could just turn into a very good way of preserving stale context.

Do you think that kind of context lifecycle — capturing, updating, and knowing when something is no longer authoritative — is part of what “persistent project understanding” needs to solve?

Collapse
 
mudassirworks profile image
Mudassir Khan

the "quietly wrong" framing is the one that cuts deepest. the bugs AI writes are not the kind you catch on a second read — they pass tests, look organized, and then announce themselves in production weeks later.

we started treating AI PR reviews the same as security reviews: assume it's plausible until you've traced the actual data path. skimming stopped being an option after a session state bug slipped through three reviewers because the code looked right on every obvious axis.

the Builders vs Coasters split is the uncomfortable part. are teams tracking reviewer depth (comment quality, bugs caught pre merge) alongside velocity, or is output count still the only dashboard?

Collapse
 
harsh2644 profile image
Harsh

Assume it's plausible until you've traced the actual data path is the security-review mindset applied exactly where it belongs, and it's a sharper standard than review carefully. plausible is the default state of AI output, that's what makes it hard to review, so treating plausibility as the starting assumption rather than a good sign flips the entire posture in the right direction.

the session state bug slipping past three reviewers is the part that should worry people more than a single missed review would. three people looked at code that was correct on every axis they were checking, and none of those axes happened to be the one that mattered. that's not a reviewer failing to pay attention, that's the code passing every test a normal review applies and still being wrong underneath it.

honest answer on your actual question: I don't know of teams tracking reviewer depth as a real metric yet, comment quality and bugs caught pre-merge aren't things I've seen show up on a dashboard the way velocity and PR count do. that's part of what makes the Builders Coasters split so hard to fix in practice, the thing that would actually reward the behavior you want is exactly the thing nobody's measuring. output count is easy to track automatically. reviewer depth basically requires someone senior manually noticing who's catching the real bugs, which doesn't scale the way a dashboard number does.

if you've seen any team actually attempt this, genuinely curious what it looked like.

Collapse
 
mudassirworks profile image
Mudassir Khan

the one attempt i saw was an issue rate — flagged issues per 100 lines reviewed, tracked per person over 90 days. fell apart fast: reviewers optimized for easy catches (nitpick rename this var), skipping the harder digs. signal became noise in about six weeks. the thing that would actually work is mapping which prod bugs were reviewable at PR stage and who was on that review. almost no one keeps that data deliberately. seen any tooling that tries to capture it?

Thread Thread
 
harsh2644 profile image
Harsh

Reviewers optimized for easy catches is Goodhart’s Law in six weeks flat. Once flagged issues became the target, flagging trivial stuff became the rational move.

A better signal is mapping prod bugs back to the PR stage: was it reviewable, and who reviewed it? That measures outcomes, not activity. I actually built a side project around this stitching git history + PR metadata to production incidents to see whether a bug was visible at review time. Still rough, but the core idea holds up.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.