AI Made Your Team Faster. Can They Still Tell You When They Are Wrong?
When execution accelerates and comprehension does not, the organisation pays in silence.
The productivity gain from AI tools is the easiest number in the building to measure. The judgment your organisation lost while gaining that productivity has no dashboard, no metric, and no owner.
Every quarter, the gap between what an institution produces with AI assistance and what it can genuinely evaluate widens. That gap surfaces when a regulatory review demands an explanation the organisation cannot give, when an architectural decision proves wrong and nobody can explain why it was made, or when an incident reveals that the people responding no longer understand the system deeply enough to diagnose it.
Two modes. One tool. Two outcomes.
In January 2026, researchers at Anthropic published a randomized controlled trial. Fifty-two junior software engineers were asked to learn a new programming library. Half used AI assistance. Half coded manually. The AI-assisted group scored 17% lower on comprehension tests. Nearly two letter grades. They did not finish meaningfully faster.
The finding has been widely covered. The detail that changes the argument has not.
The researchers found that how someone engaged with the tool determined what they retained. Engineers who used AI to ask conceptual questions, to understand why the code worked the way it did, scored 65% or higher on comprehension. Engineers who delegated code generation to the tool scored below 40%.
Same tool. Same task. Two modes of engagement. Two very different outcomes for the organisation that employs them.
In the previous post, I described two modes of thinking that appear wherever technology decisions are made. The System SME iterates on what exists. They know the current system, they can extend it, they can make it do more of what it already does. The Architect questions whether the work belongs where it is being placed. They evaluate the output against the problem space, not the immediate requirement.
The Shen and Tamkin study did not use those terms. However, the parallel is direct enough to be worth drawing. Delegation mode is System SME mode applied to a tool. Accept the output. Iterate. Move on. Conceptual inquiry mode is Architect mode applied to a tool. Interrogate the output. Understand the reasoning. Develop the capacity to evaluate whether what the tool produced is correct.
The institutional question is which mode the organisation is designing its AI deployment to amplify.
Almost every deployment I observe amplifies System SME mode. Engineers are given tools to produce faster. Architects are not given tools to evaluate better, to test assumptions at speed, to model consequences before they compound. The investment goes entirely into acceleration. The judgment that makes the acceleration trustworthy receives no equivalent investment.
The supervision loop
The reasonable challenge is that we have accepted comparable trade-offs before. Calculators replaced mental arithmetic. Satellite navigation replaced knowledge of routes. In each case, the skill loss was absorbed and the world carried on.
The difference is the supervision loop.
A calculator removes arithmetic. You do not need arithmetic to verify the calculator’s output. You check the number. Satellite navigation removes route memory. You do not need to remember the route to verify you have arrived.
AI-assisted engineering and architecture remove comprehension and judgment. You do need comprehension to evaluate whether AI-generated architecture is sound. You do need engineering judgment to supervise AI-generated code at scale. The skill being displaced is the same skill required to verify the output.
When the supervision loop is broken, the organisation has no reliable way to distinguish good output from output that merely looks good.
The speed continues. The quality signal degrades. The people who would have noticed are the ones whose judgment was never developed.
The trajectory
The speed is accelerating faster than the judgment.
In June 2026, Anthropic’s research arm published data showing that more than 80% of the code merged into their own production codebase is now authored by their AI tool, Claude. The typical engineer merges eight times as much code per day as in 2024. The same paper noted that large performance gaps persist when it comes to the AI exercising judgment in choosing goals. This is vendor self-reporting. However, the pattern it describes is visible across the industry.
Anthropic named the underlying dynamic in an earlier internal study of 132 of their own engineers and researchers, conducted in August 2025. They called it the paradox of supervision. Effectively using AI requires supervision. Supervising AI requires the very skills that may atrophy from AI overuse. A senior engineer in the study observed that they use AI primarily in cases where they already know what the answer should look like, a capacity they developed by doing engineering without AI assistance. Their concern was for those earlier in their careers who may never build that capacity at all.
At the 2026 MIT Sloan CIO Symposium, the term “AI atrophy” entered the vocabulary of senior technology leaders. The conversation is happening. However, it is framed as an individual concern. How do we keep our people sharp? How do we encourage critical thinking?
The structural question sits one level above. How does the organisation design its AI deployment so that judgment is accelerated alongside execution?
The pattern I have seen before
I have been inside this pattern before AI tools arrived.
When I proposed Domain Driven Design for a significant new platform build, there was substantial resistance. Various parts of the organisation had attempted DDD in the past without success. The resistance was earned. The understanding of what a domain model required, and the readiness to work within one, was not where it needed to be.
The earlier attempts had adopted the “what” without investing in the “why.” Teams could draw domain boundaries. They could not explain the reasoning behind them. They operated in System SME mode on the domain model itself. They inherited the structure without developing the judgment to question it or adapt it when the context changed.
The work that made the difference was years of deliberate investment in helping people understand why a boundary sat where it sat, why a capability belonged in one domain and not another, why the reasoning mattered as much as the result. That is Architect mode applied to an organisational transformation. Building the capacity to evaluate the model, challenge it, and carry it independently.
After multiple years, the platform build succeeded. More importantly, the organisation started adopting the approach more widely without me carrying it.
That independent adoption was the proof. The judgment had transferred.
The organisation could hold the model, question it, and adapt it without the person who introduced it in the room.
AI tools are following the same path at higher velocity. Organisations are adopting the “what.” They are deploying tools, measuring output, celebrating speed. Very few are investing in the “why.” Very few are designing their deployment so that the people using the tools develop the judgment to evaluate what the tools produce. The failure mode is the same. The velocity makes it harder to catch.
Both. Not one at the expense of the other.
The answer is not slower adoption.
The Shen and Tamkin data shows the path. The tool supports both modes of engagement. Conceptual inquiry, Architect mode, retains comprehension and develops judgment. Delegation, System SME mode, erodes comprehension and accumulates dependence.
What does designing for Architect mode look like in practice? It means requiring teams to articulate the reasoning behind AI-generated output before accepting it. It means using AI to generate multiple architectural options and having the team evaluate trade-offs rather than taking the first output. It means treating AI deployment as a transformational programme that invests in the “why” alongside the “what,” the same investment that determines whether any major change, DDD, platform migration, operating model shift, succeeds or calcifies into something nobody can explain.
The organisation that designs its AI deployment to accelerate Architect mode alongside System SME mode gets both speed and judgment. Engineers who use the tool to interrogate, to understand, to question retain the capacity to evaluate what the tool produces. Architects who use the tool to test assumptions, model consequences, and pressure-test designs gain speed without losing the judgment that makes the speed safe.
The organisation that designs only for System SME mode gets speed. And an invisible deficit that widens until it becomes visible in the worst possible way.
Every technology leader deploying AI tools is making this choice. The question is whether they are making it deliberately.
Speed without comprehension is a debt the organisation repays under pressure.
If you found this useful, the likelihood is someone you know is asking the same question. Pass it on.