This week the Hacker News front page handed 457 points and 216 comments to a post on lynchlab.net with a headline built to start a fight: a ten-year-old convolutional network, it reports, outperforms frontier LLMs at skin cancer diagnosis. The thread reads exactly as you’d expect. Half of it is people who have been waiting three years for permission to say “told you so.” The other half is people explaining that the comparison was never fair.
Both halves are having the wrong argument.
A 2016 Network Still Tops the Table
Take the result at face value. A model designed around 2016 — before transformers ate the world, before anyone said “frontier” without a small smirk — beats the current general-purpose models on a narrow image task.
The lazy reading is that convolutional networks are secretly immortal. The economical reading is that the table hasn’t moved.
Benchmarks age. That is what they are for: a fixed target so that progress stays legible and comparable across papers. But a fixed target is also a target everyone can spend ten years aiming at, and after a decade the aiming gets extremely good. The number climbs. The question underneath the number — does this thing work on the patients who actually walk into a clinic in a county with no dermatologist — goes unasked, because the benchmark never asked it.
So the honest inference from a decade-old architecture winning is not “old is better.” It is that the scoreboard has sat in the same place long enough that the gap between a frontier model and a 2016 workhorse is now smaller than the gap between either of them and clinical reality. That is a much less satisfying sentence than the one in the headline, which is presumably why nobody wrote it.
What a Frozen Metric Buys
A stable benchmark is a wonderful thing to own. It is publishable. It is citable. It is defensible in a procurement meeting when a hospital board member asks why the system is spending money on software instead of a nurse. It has, in a word, a career.
A dermatology resident I spoke with in a basement reading room between clinics put it more plainly than any journal editorial has: “We get a paper every month with the same number on it. Nobody has ever shown me what happens when the thing is wrong.”
That is the sentence that should sting. Not because a chatbot lost a bake-off, but because the apparatus that was supposed to tell her whether any of this works has been grading the same exam since she was in high school. A frozen metric does not merely fail to measure progress. It actively launders staleness into authority — and the longer it runs, the more citations it accumulates, until the number is no longer evidence of anything except that a lot of people have cited it.
The Losing Model Gets the Contract Anyway
Now the part that should make the vindication crowd uneasy.
Suppose the skeptics are entirely right. Suppose the small, cheap, narrow, validated model is the correct tool and the general model is a party trick with a demo budget. Does the correct tool get deployed?
Not obviously. A lesion classifier is a feature. You cannot build a platform strategy on it, you cannot sell it to twelve departments, and you cannot put it on the cover of an annual report as a transformation. The general model that just lost the benchmark can be sold to radiology, to the ED, to compliance, to the call center, and to whoever else signs the invoice. Benchmarks do not decide procurement. Addressable surface area does.
The hospital did not buy the best model. It bought the one with the most departments attached to it. That is not a failure of the technology and it is barely a failure of the regulators — it is a failure of the buyer’s incentives, and it will survive any number of humiliating leaderboard results. The boring winner stays in the lab while the loser gets the contract, and everyone involved writes a press release about rigor.
A Benchmark With an Expiration Date
The fix is not more leaderboards. We have leaderboards. The fix is a norm that a benchmark is a perishable good: publish it, use it, retire it, and force the field onto fresh data and fresh sites before the citations pile up. External validation on held-out hospitals, not held-out rows from the same archive. Prospective, not retrospective. The unglamorous stuff that no one gets a keynote for.
That would be worse for the incumbents and worse for the challengers and better for the resident in the basement.
This is an argument about how we grade machines, not about anyone’s skin, and the two should not be confused. But the confusion is precisely the product being sold.
The people who should feel least comfortable this week are not the model vendors. They are the ones who have been citing the same number for a decade and calling it evidence — and the ones waving this post around as proof that the boom is over, who have noticed only half of what it shows. A decade of frontier money bought nothing for this task. A decade of benchmark stewardship bought nothing either. The old model didn’t win. Nobody did.