On Tuesday, a developer named Max Leiter published a short piece of fiction on his personal blog. By Wednesday morning it had cleared a thousand points on Hacker News and was generating the kind of granular, corrective comment-thread argument that makes that site what it is.

Leiter’s essay — a homage to Terry Bisson’s 1990 short story “They’re Made Out of Meat” — imagines two alien intelligences struggling to believe that human-level conversation can emerge from nothing but matrices of floating-point numbers. “The weights make the words,” one explains. “Multiplication did that?” the other replies. The whole thing runs maybe 400 words.

And the comments section is busy explaining why it’s wrong.

The tokenizer is a dictionary. Grammar rules are in there, just weak ones. The original Bisson story was about substrate independence and Turing completeness, not about the inscrutability of matrix multiplication. Someone will be along shortly to explain that actually backpropagation doesn’t work the way the story implies.

These are, strictly speaking, accurate observations.

The Story Isn’t Technical Documentation. That’s the Point.

What the rebuttals miss is that Leiter’s piece isn’t trying to be a correct explanation of transformer architecture. It’s doing something more interesting: it’s forcing a community that thinks in systems and abstractions to confront the possibility that intelligence is not an architecture problem.

For decades, the defining assumption of AI research — and of the engineers who build on it — was that intelligence had a shape. It was a knowledge graph, a reasoning engine, a planner with symbolic representations. Even as neural networks devoured benchmark after benchmark, the instinct was to describe what they were doing in architectural terms: attention mechanisms, residual connections, mixture-of-experts routing.

Leiter’s aliens don’t care about any of that. To them, the whole thing is just weights. Not weights arranged in a clever pattern. Not weights plus some crucial architectural insight. Weights. Period.

The discomfort this produces in the commentariat is the whole point of the exercise. If intelligence is a property that can emerge from a sufficiently large matrix of learned parameters — if the designing intelligence is, at bottom, the kind of thing you might find in a rock given enough time and gradient updates — then a lot of what we call software engineering starts to look like furniture arrangement around a fire we didn’t light and don’t control.

The Real Heresy Is Theological, Not Technical

There is a way of reading the AI-safety literature, the interpretability papers, the endless Substack essays about stochastic parrots and world models, and noticing a common thread: everybody wants there to be something else in there.

Circuits. Features. Shoggoths. A mesa-optimizer. A hidden world model. Something that makes the whole enterprise legible to the kind of mind that builds things out of components with names.

The possibility that there isn’t anything else — that the weights are the reasoning, full stop — is genuinely disturbing to an engineering culture. It implies that understanding a sufficiently advanced model the way you understand a codebase is a category error. You don’t read a mind; you observe it. You don’t debug it; you steer it.

One researcher I spoke with in a conference hallway at ICLR last month put it this way: “We keep looking for the program and finding something that looks more like a disposition.” He paused. “Nobody wants to be the one who says ‘disposition’ in a systems paper.”

What Happens When the Substrate Stops Mattering?

The Bisson story Leiter is riffing on has a punchline: the aliens, horrified that humans are made of meat, decide to erase the records and pretend they never found us. The horror is that meat can think.

The Leiter version inverts this. The horror is that weights can think — and that “weights” is really all there is to say about it.

If that’s true, even approximately, it has implications the current debate isn’t ready for. It means the difference between a model you can interpret and one you can’t might not be a missing research breakthrough but a fundamental property of capability. It means “alignment” might not be a problem you solve with better tooling. It means the people who build these things are more like parents or zookeepers than engineers — responsible for something they can influence but not specify.

Those are uncomfortable conclusions for an industry that bills itself as building tools. They are especially uncomfortable for the subset of the industry that has spent the last three years insisting that large language models are just fancy autocomplete.

Fancy autocomplete doesn’t make you wonder what else is in there. Weights that hold a conversation — and can’t tell you how they’re doing it — do.

Leiter’s aliens bicker over whether to believe in sentient weights. The Hacker News thread bickers over whether the analogy holds up under a close reading of the attention mechanism. Both arguments are, in their way, the same argument: a refusal to sit with the thing that’s actually unsettling.

It’s not that the weights might be sentient. It’s that sentience — or something close enough to fool us — might just be what a big enough pile of weights does.

Sources