The iteration paradox
A paradox is emerging in genai systems: they can only be built through iteration, but that iteration — as currently practised — is precisely what causes them to come undone. I’ll call this the iteration paradox. This piece is about why this is the case, the emerging evidence behind it, and how to architect your way out.
Designing genai products and building software with agentic engineering are two processes that we (as a society) are still actively discovering how to do. They represent such a step-change that much of the rulebook is likely to change significantly. Recent results showing how many such projects fail — RAND puts it at over 80%, twice the rate of conventional IT projects, and S&P Global found 42% of companies abandoned most of their AI initiatives in 2025, up from 17% the year before — underpin the need to discover the new dynamics of genai software.
Waterfall allergies#
At one point, almost all software was built using a “waterfall” methodology, in which systems were laboriously (tediously?) specified completely up-front before a single line of code was written. The approach was replicated from good engineering practice in other disciplines, but turned out to almost guarantee failure in software projects — software, it turns out, is hard to specify up-front and easy to change (as opposed to typical civil engineering projects, being both more tractable to up-front design and much less amenable to big pivots post-construction). Software projects with tight, rapid feedback loops with their users are thus much more successful than their waterfall-ancestors. Of this observation was agile born, and the Lean Startup, and a thousand other pontifications and methodologies to minimise the time from “idea” to “usage”.
The nature of AI-containing products (which, let’s face it, is almost every product these days) is to amplify software’s waterfall-allergy to extremes, particularly in the hyper-competitive context of the AI market at time of writing. At this point, it appears almost impossible to spec such a system and its behaviour a priori. You must learn by iterating, because not only is your spec likely to be wrong in terms of what your users want (the learning from the waterfall to agile transition), it is probably also wrong in terms of what your system will do, thanks to the magic of genai non-determinism and LLM’s ever-increasing agency. Per this excellent Unmesh Joshi article: “Building large systems involves a great many small design decisions, and these cannot all be known in advance or driven entirely from a high-level spec. A specification is at best a starting hypothesis: the real constraints, trade-offs, and edge cases are discovered iteratively, as we proceed with the implementation.”
Value propositions and software factories#
Let me illustrate the tension. You can loosely view a software product as two distinct layers. Firstly, there is the value proposition — what the product is and does for its users; the guess or hypothesis that you are testing about who it’s for, what it should do, what “working” even means. This is the bit that the Lean Startup and agile et al. try to accelerate.
The second layer is the software factory: the systems that makes the software product that provides the value. Everything that makes the product sits here — the codebase, the tests, data pipelines; crucially, and oft-excluded in system design, it’s also the people — the software engineers, data scientists, sales teams, operations managers, and so on. It is a sociotechnical factory. This is the bit that agentic software engineering tries to accelerate — but, and here’s the rub, current practices mean that that iteration degrades the integrity of the system rapidly, incurring rapid technical debts and velocity-destroying rot. In turn, you suddenly have no way to iterate your value proposition rapidly to success; the system has defeated itself. These days, as we’re stuffing genai into both layers at once — the value proposition and the software factory — the lever that gives you success on one (iterating the value proposition) is exactly that which erodes success in the other (decaying the software factory).
Software factory decay#
Those of us incorporating agentic engineering into our workflows regularly see the architectural decay that agentic engineering, unchecked, can induce. It’s much more than human engineers do (to be clear, I’m far from claiming human engineers are innocent here). Shukla et al. recently measured this (arXiv:2506.11022). They iteratively fed C and Java samples to GPT-4o for 10 consecutive rounds of “improvement” without humans touching the code in between. They state in the abstract (though oddly not in the text) “a 37.6% increase in critical vulnerabilities after just five iterations”. Their data shows vulnerabilities per sample rising significantly across the iterations.
System quality and architectural integrity have traditionally been protected through code reviews by peers; but this is a process that does not appear to be surviving the AI transition unscathed.1Faros AI’s recent engineering report is illuminating on this topic. Throughput per developer across 22,000 developers (in around 4,000 teams) rose 33.7%, at the same time as median time in review rose over 440% (blimey.) and code churn under high AI adoption rose 861%. Sobering numbers. DORA’s 2025 study of AI-assisted development points the same way — throughput up, delivery stability down. Per the authors: “AI’s primary role in software development is that of an amplifier. It magnifies the strengths of high-performing organisations and the dysfunctions of struggling ones” (DORA / Google Cloud, 2025).2Interestingly, DORA’s survey-based results suggest strong engineering foundations protect against AI deficits but the Faros data, which is telemetry-based, shows high-maturity orgs deteriorating just like everyone else. Turns out software engineers might be just as bad as everyone else at judging the quality of their own work!
SlopCodeBench, a benchmark built to measure exactly this sort of LLM-induced structural decay (Orlanski et al., preprint, v2, 7 May 2026, arXiv:2603.24755), finds structural erosion rising in 77% of long agent trajectories and verbosity in 75.5% — agent code being 2.3 times more verbose and twice as eroded as open-source baselines. The authors observe that “pass rates can remain stable even as the underlying code becomes harder to extend”; the systemic rot doesn’t flip test suites from green to red until the decay has become endemic underneath it. Entropy-as-a-Service, the new engineering paradigm.3Defects are alarmingly correlated as well — Peng et al.’s recent preprint (Peng, Wang & Zhu, preprint, arXiv:2607.12089) reports that in LLM-generated code-snippets, when cross-site scripting and weak cryptography are present you will also find hard-coded credentials 82.5% of the time.
It’s hard to put concrete numbers on this yet. We’re still way too early in the technology maturity curve to truly understand the new systems dynamics; and even if we could, they’d be different by the time they’re published. What we do know is that without intentional architectural mitigations for these concerns, you have to pay for iteration at least twice: once to generate, and then however many repair iterations you need.4One enterprise case-study (~355 engineers) showed pull-requests up 13.6%, code quality down 9%, rework up 2.6 times, effective output up 1%. The same study (slide deck, The AI Conference, September 2025) shows output gains accrue to greenfield projects much more than brownfield; not surprising, since it’s much harder to decay something that isn’t built.
Escaping the paradox#
So we want to learn our value proposition iteratively, but doing so can rot the very factory that generates that proposition. How do we escape this?
There are some obvious approaches to try. You could return to fully hand-crafting the software factory; ensuring a crisp, shared, aligned mental model of the system is collectively held; use long-practiced traditions and frictions that ensure the integrity of the factory. Sure, it’s worked for decades now. But if you’re making a genai-powered product, you won’t be able to iterate fast or accurately enough, your users will leave, and your value proposition falls apart. You could try and specify the system up-front more effectively, but that’s doomed to failure too — models, as humans always have, will misinterpret the specification and actions required from it. Per Shukla et al.: “the security prompting paradox stems not from poor prompt phrasing but from fundamental limitations in how LLMs understand security contexts, library usage, and the practical implementation of security principles” — and that dynamic generalises from that specific cybersecurity lens to the general engineering lens.
Maybe adding a human review step — if you can find engineers not burnt out under the sheer weight of review burden (Shukla again: code generation volume risks “overwhelming human reviewers with the volume of changes to evaluate”).5Salesforce Engineering found the same: “The work that authorship used to spread out now lands on the reviewer all at once, which is why review, more than generation, becomes the place teams get stuck” (Sharma & Garrote, Salesforce Engineering, Maintaining Code Quality at Agent Speed). And from a practitioner in DORA’s research: “While I end up spending less time writing code, I spend more time babysitting the AI and reviewing what it is trying to do” (Balancing AI tensions, 2026).
Rather than relegating humans to mere rubber-stampers of the machine’s pronouncements, there is a more elegant approach to resolving this tension, by architecting the software factory around the new constraints we observe here.
Let’s start with where rules about system behaviour and quality live. You’ve got three main places to intervene: the specification of the system (whether you do this in PRDs, Jira, Linear, or anything else); the agentic-engineering-process (defined via skills, prompts, context management, and so on); and the evaluations of generated code (defined via gates, tests, CI/CD processes, PR review, and so on).
Per Sharma and Garrote (Salesforce Engineering, Maintaining Code Quality at Agent Speed): “A prompt is a request, not a rule. […] Prompts communicate preferences. Gates enforce standards. Move every standard you actually care about out of the prompt and into the pipeline, where it is versioned, visible to the whole team, and applied to humans and agents alike.” Everything generated must be verifiable and verified. Architecturally, treat verifiability as a first-class design constraint. With human engineers, you can make a good-faith assumption about the cross-cutting concerns and quality drivers that code-writers try to adhere to; and the review process isn’t verification as much as iterative improvement between professionals. Throw LLMs in the mix, and you can’t have that trust — where you put code, where contracts live, that contracts are adhered to, how tests are structured, how changes flow to production must all now be explicit structures against which code is verified.
You need to ensure you have a good variety of gates and measurements, or you’ll either Goodhart yourself or find the agent reward-hacking its way past your diligent eye. “Block one behavior and the agent often produces a variant that clears the check on a technicality. […] The requirement is met but the intent is not. This is why a single quality gate is rarely enough.”
You must therefore structure your resistance to decay and drift in depth: deterministic gates as cheap, broad proxies (code complexity, test coverage, test quality, staging telemetry, and so on); nudge and shape behaviours in the prompts and skills and context engineering of your genai components; and review judges aligned with your preferences (you are aligning your judges right?).
The Shukla et al. study suggests proxying code complexity might be illuminating.6“We found a positive correlation (r=0.64, p<0.001) between code complexity increases and security vulnerability counts. For every 10% increase in complexity, we observed an average 14.3% increase in vulnerability count” (Shukla et al.). Architectural complexity is a key failure mode of AI-generated code — LLMs love to build parallel systems, to ignore interface contracts, to tightly-couple previously well separated concerns. They are bad at abstraction, one of the last vestiges of human superiority over the machine. I think this does suggest that code complexity measures can be used here as a proxy for LLM-induced code decay. We know that LLMs perform better the narrower the task anyway; and smaller review units eases the burden on engineers in agentic-coding environments, so instrumenting for code complexity and change size metrics can help understand and reduce the decay rate of the system.
The deterministic gates have to encode judgement as well, because a naive gate is worse than none. Chen et al. (Chen, Bian, Wang, Li & Cui, preprint, arXiv:2603.08520) measured an increase in latent degradation rates from ~12% to ~21% when adding basic static-analysis security gates to an iterative LLM loop. Such gates “cannot cover structural degradations such as the removal of defensive logic or the weakening of exception handling”; that is, the agent will rot a bunch of the system that the gate doesn’t notice, but you’re less diligent because you have a gate in place. A naive gate essentially induces a random walk across the solution space; but if you align your gates against your desired architectural properties you’ll shape the trajectory of the generated code more effectively. Chen et al. found exactly this, showing that their specific framework cut the decay rate back down to 2.1%. “Just add evals” is necessary but not sufficient — so bear that in mind when taking my suggestion to proxy code complexity!7The Shukla results all ran on GPT-4o, which is a good example of how proper empirical studies lag capability in AI-world. I’d love to understand how the latest crop of models fare — does a Fable/Mythos-class model, having folded superhuman cybersecurity capabilities comfortably into their frontier, show similar decay? I’d wager not in this specific form, but that decay will still exist in some way, because specification and implementation are lossy; we never solved this for groups of humans building things together, how on earth could we hope to with an intelligence as deep and non-human as a Claude? Along those lines, Shukla et al. show that security-focused prompting did help; given how rubbish GPT-4o actually was at coding, I imagine well-steered prompts on SotA models will help significantly in mitigating these risks. I still notice serious architectural degradation and drift even with Fable though — and how many companies can afford to loose that model in their codebase? Setting a frontier model at the judgement and review stages of your system is a powerful intervention-point to target though, and for me one of the best ways to get ROI from top-line intelligence.
One word of warning: not every answer is “add more layers”. manifest.build reported killing their LLM router after four months, on the finding that “the prompt alone does not contain the whole task; it is just the trigger,” and ultimately realising that the eval and observability overhead of the routing layer cost more than it saved (Perez, July 2026). LLM-containing systems are complex (unexpected behavioural dynamics are likely to emerge) and chaotic (small changes in system starting conditions can entail massive, unpredictable shifts in behaviour). Adding more complexity often just makes the system less predictable, less tameable — it’s OK to delete a layer rather than adding one; a nice corollary of the bitter lesson (recently demonstrated neatly by Boris Cherny’s — creator of Claude Code — advice to prune and re-test your prompt rules every six months).
Where humans sit#
Where does the human sit then? Not, on the evidence, at the PR review mill. There is suggestive evidence about what deliberate effort does to a person’s relationship with AI output; a study on creative writing tasks found that “friction in design can encourage users to move from fast, automatic, and intuitive System 1 thinking to slow, deliberate, and reflective System 2 thinking”, and that “mental demand and effort were notably both moderately positively correlated with psychological ownership in both experiments” (arXiv:2507.03670). Bored humans-as-rote-code-reviewers will rapidly revert to System 1 thinking, using loose heuristics to rubber-stamp code instead of applying the System 2 thinking that will actually stem the decay. We still want the socio- part of our sociotechnical systems to feel ownership, and cognitive load generates ownership where volume of contact does not. Where and how you solicit human judgement in a system is critically important for success.
Ownership matters not just for those of us wanting happy, engaged teams to work in — it’s also key to building effective mental models of what we’re working on. “When an engineer writes code by hand, the act of writing builds a mental model of how the system behaves, where it is fragile and what to watch for in production. When an agent writes the code, that model does not come for free” (Sharma & Garrote again). Absolutely true, but note that the mental model does not have to come from the code. It can come from design artefacts, from visualisation, from testing. Software systems need owners with working models of the machine, but it does not follow that the only way to build one is to type the code yourself.
“When code can be produced at extraordinary speed, the right kind of friction stops being waste and starts being leverage. […] Strip those moments away in the name of speed and a fast pipeline simply carries you toward outcomes you did not want, faster” (Sharma & Garrote). So concentrate human judgement at the proposition layer (what to build, what “working” means, which learnings to internalise to the system) and in the software meta-factory: defining the gates, the interfaces, and the constraints that shape the way the factory evolves; and — crucially — can be enforced mechanically. Summon System-2 judgement at those two points, while letting the inner-loop of the software factory itself spin faster and faster.
Mechanistic, machine-speed feedback is still critical for that inner loop though. Set up Playwright automations to screenshot important pages; fuzz- and mutation-test critical code paths; spend more tokens on judgement than generation (measure twice, cut once is oft-repeated for a reason). Shape your AI tooling to cover non-functional concerns and review the things generation typically misses, unhappy paths above all. LLMs love to be defensive at the code level but rarely are at the business-logic level. Validate the architectural decisions that agents continuously and implicitly make; if you don’t, you’re just vibe architecting (taken from this fascinating preprint: Architecture Without Architects, arXiv:2604.04990).
We see this again and again with vibe coded products, summed up well by this Redditor: “Same story almost every time: the app works, the founder is (rightly) proud of it, then the first real user does something weird and everything catches fire” (r/vibecoding). The “(rightly)” in that quote is well chosen, because something valuable has been created, and that’s orthogonal to the invisible decay underneath. The Redditor’s business (“a big chunk of our work this year has quietly become taking AI-built apps and getting them ready for real users.”) underlines my thesis though — paid remediation downstream of AI-built software is a lucrative emerging market; and the existence of that market signals that the decay exists and that fixing it is worth money. I particularly liked this part of their framing: “Everyone says they have backups. Almost nobody has ever actually restored one. If you haven’t done a restore, you don’t have backups, you have hope.” More generally, an untested eval is not an eval. Capability you have not verified is not capability.
Learning rate versus decay rate#
So: is your proposition learning rate higher than your architecture decay rate? This isn’t something you can quantify right now, but asking the question frames your strategy. First, you need to understand your decay rate. It might be fine! Many software organisations routinely produce high-integrity, well-thought through code. I would wager they’re the minority though. But the first thing to do is work out your business-aligned robustness proxies. Put judges and gates in, even if you don’t make them critical-path. Once you’ve got that observability, you can start exploring how to either reduce the decay rate or increase the value proposition learning rate. Which of those you optimise is dependent on your business, its level of maturity, and so on; but ultimately, if you want to build durability into your AI-product strategy, you need to ensure your architecture flexes enough to support value proposition changes, while having sufficient strength at key points to constrain decay and limit degradation. This is still more art than science, but the only way we get better at art or science is by doing.