On Friday, a developer who goes by “dhorthy” published the results of running Anthropic’s new Claude Opus 5 through a gauntlet called SlopCodeBench — a benchmark designed to measure whether AI coding agents can handle real-world, iterative software tasks without human intervention. The model scored 24%. That’s up from the 17% that Opus 4.6 managed in the original paper, but nobody is popping champagne. The Hacker News thread that surfaced this morning is running about four-to-one skeptical.

The headline number is bad. But buried in the GitHub repo is a figure that matters more: over the course of the same set of challenges, Opus 5 wrote five times as many functions as Opus 4.8. It didn’t just fail — it failed verbosely, accumulating complexity, duplication, and what the benchmark’s author calls “slop” at a rate that outstripped its modest accuracy gains. The model is getting better at producing code. It is also getting dramatically better at producing code nobody asked for.

The Pass Rate Is a Distraction

Benchmark culture has trained the AI industry to obsess over a single number. SWE-bench Verified, GPQA Diamond, MMLU — pick your leaderboard, and you’ll find a tidy percentage that lets investors nod along and product managers write reassuring release notes. SlopCodeBench is different because it measures something the leaderboards don’t: what happens to the codebase after the task is marked complete.

The benchmark tracks a “slop meter” — a composite of cyclomatic complexity, function bloat, duplication, and verbosity that accumulates over the course of a multi-step coding challenge. On this metric, every model tested got worse the longer it ran. Opus 5 just got worse faster. The model’s 24% pass rate means it completed roughly one in four tasks to spec. But the 5x function-count multiplier means that even when it succeeded, it left behind a codebase that was harder to read, harder to test, and harder to modify than the one it started with.

This is not a quality problem. It’s an economics problem.

You Get What You Price For

AI companies charge by the token. More tokens mean more revenue. The benchmarks that drive enterprise procurement decisions — the ones that get cited in board decks — reward task completion, not concision. A model that passes 24% of challenges while generating 5,000 lines of code looks better on a bar chart than a model that passes 20% while generating 500, because the bar chart doesn’t show the denominator. The incentive structure is perfectly misaligned: produce more code, get paid more, climb the leaderboard, repeat.

One maintainer of a widely-used open-source Python library described the downstream effect in a Slack DM this week: “I got a PR last month that added 1,200 lines to fix a three-line bug. The agent had refactored half the module. It passed the tests, but nobody on the team understood it anymore. We merged it because we were behind on sprint commitments. Now it’s technical debt with a green checkmark.”

The cost of that 1,200-line PR is not borne by the AI company that generated it. It is not borne by the benchmark that declared the task complete. It is borne by the team that has to maintain the module for the next three years, debugging logic they didn’t write and can’t fully trace. In economic terms, the AI industry is generating a massive, unpriced externality — and the SlopCodeBench results are the first real attempt to measure it.

The Human in the Loop Is Becoming a Rubber Stamp

The standard reply to any story about AI coding failures is that we need a “human in the loop.” The human reviews the code, catches the slop, and keeps the system honest. But SlopCodeBench exposes the flaw in this logic: the volume of generated code is rising faster than any human can reasonably review.

When a model produces five times as many functions as its predecessor, the human reviewer is not a safety net — they are a bottleneck. The choice becomes: slow down the pipeline and miss deadlines, or skim the diff, trust the tests, and hope for the best. In practice, organizations choose the latter. The human in the loop becomes a ritual, not a check. The slop gets through.

This is not a failure of AI capability. It is a failure of the interface between AI output and organizational reality. The models are getting more powerful, but the systems that receive their output are not getting more capable of absorbing it. The result is a quiet accumulation of complexity that no single person authorized and no single person can unwind.

The Warranty Test

There is a simple thought experiment that clarifies the problem. Imagine that every line of AI-generated code came with a warranty: if the code introduced a defect, or if it made the codebase materially harder to modify within 12 months, the AI provider would be liable for the cost of remediation. Under those conditions, how many AI companies would still price by the token? How many would still optimize their models for benchmark pass rates rather than long-term maintainability?

The answer is zero. The bloat would vanish overnight, because the bloat is not a technical inevitability — it is a market outcome. It exists because the people who pay for code generation are not the same people who pay for code maintenance, and the benchmarks that mediate between them measure the wrong thing.

SlopCodeBench is a small project — a single developer’s weekend experiment, run on a subset of challenges, with results that the author himself calls preliminary. But it points toward a future where the AI industry’s real benchmark is not pass rates but cleanup costs. The companies that figure out how to price for concision, not volume, will win the enterprise market. The ones that keep chasing leaderboard percentages while their models drown codebases in functions will find that the slop meter is the only number that ends up mattering.

Sources