"Applying sound methods to stupid projects"

Adam remains one of the most engaging writers in my feed.

This post covers a lot of ground, but it does a great job of highlighting that while AI has been developing intelligence quickly along one axis, there's another branch of intelligence that current methods still haven't conjured in these models.

This is one reason skeptics think the approach the frontier labs are pursuing is unlikely to lead to Artificial General Intelligence (or the Singularity, or Superintelligence, or whichever label and loose definition you prefer).

Here’s what I highlighted:

Some problems have clear boundaries and verifiable solutions, like “What’s the cube root of 38,126?”. These problems require objective intelligence. Other problems are vague and squishy and it’s not clear whether you’ve solved them, or whether they exist at all, like “How do I live a good life?”. These problems require subjective intelligence. Objective intelligence can be trained, reinforced, and validated. Subjective intelligence cannot.

It’s unfortunate that people use one word to refer to both of these capabilities, when in fact they have nothing to do with each other. It is also, ironically, a case of objective intelligence overshadowing subjective intelligence: these skills are obviously and intuitively different, but a century of psychological research has “proven” that only one of them exists.

The problem is that any test of intelligence is only ever a test of objective intelligence. “How do I live a good life?” is not a multiple-choice question. “Discovering” the g-factor again and again is like being surprised that you find the same patch of sidewalk every time you look under the same streetlight.

AI is pure objective intelligence. That’s why each new model comes with a report card instead of a birth certificate

The promise of artificial superintelligence is based on the idea that objective intelligence is the only intelligence. Or, even if there are multiple forms of intelligence out there, that they are fungible. To be an AI maximalist is to believe we are playing under Settlers of Catan rules, where if you have enough of any one resource, you can trade it for any other resource. If you have infinite objective intelligence, then you have infinite everything.

It’s hard to describe exactly what the machines are missing. Have you ever loved someone who once loved you back, then didn’t anymore? Did you notice how their eyes dimmed? Did you note the disappearance of that subtle wrinkle in the temples that distinguishes a real smile from a fake one? Did you catch it when you stopped being cared for and started being humored? The moment you realize what’s happening, you age out of your enchantment—one day you’re crawling through a wardrobe to Narnia, and next day you open up the wardrobe and there’s nothing but hangers. Talking to an AI feels a bit like that, except without the nice part at the beginning.

The words themselves don’t contain that feeling—they are a recipe for creating that feeling inside your own head, to assemble the right set of emotions out of the experiences you have at hand. If I do a good job, the subjective experience that results inside you might resemble the one that originated inside me, but it will never be identical, because we’re working with different ingredients.

The computer doesn’t know any of this. It can’t know any of this. It can only read the cookbook; it can’t taste the meal. Objective knowledge can make your sentences true, but it can’t make them alive. Without access to subjective knowledge, you quickly hit a wall. And unlike all previous walls that AI has surmounted, you can’t overcome this one by scaling—either in the literal or metaphorical sense—because it’s a wall with a width you cannot describe and a height you cannot see.

The fact that it’s hard to describe how to improve AI writing is, of course, the exact problem. You can’t put a number on the things it does wrong, and you can’t minimize what you can’t measure. That’s the wall.

I find this very fortuitous, of course, but I also find it pretty funny, because me vs. the machines should be no contest at all. I have not read the entire internet or even that many books. I do not have a team of Stanford PhDs working round the clock to make me better at my job. Nobody has invested $2.5 trillion in me. I should be lying dead somewhere in West Virginia, my heart burst open after losing to Claude Opus 4.6 in a John Henry-style showdown. Instead, I get to write my little posts because nowhere, in all those data centers, are the specific thoughts that happen to occur in the dumb hunk of meat ensconced in my skull.

I think all of us suffer from this bottleneck blindness: we assume our current bottleneck is our only bottleneck. When you’re strapped for cash, you think all of your problems are cash problems. But once you’ve got some money in you pocket, you realize that what you really need is time. Free up some time, and you discover that you’re actually lacking motivation. Acquire some motivation, and you realize what you’re missing is ideas. Then you need direction, then you need discipline, then you need buy-in, and so on, forever.

Well, once you get access to an infinite Madame Stats and Mr. Encyclopedia, you realize they can’t get you very far. For one thing, you can’t rely on Madame Stats and Mr. Encyclopedia entirely, because if you can’t do any stats and you never read any papers, you’re probably not going to have many interesting ideas yourself.⁵ Plus, while the Stats/Encyclopedia duo can tell you whether your experiment has been done before and whether you’ve run the numbers correctly, they can’t give you the single most important piece of feedback: they can’t tell you whether your idea is boring.

The most important thing I learned during my PhD was how to be bored correctly. Novices think everything is exciting, or they think everything is boring. Only masters are bored by the right things.

I don’t say this as someone who is allergic to the idea of AI, or who has only spent 15 minutes screwing around with a single model, hoping it will do something stupid so I can go tattle on it. If the talking computers said lots of fascinating things, I don’t see any point in trying to tell a noble lie about it. And if AI can cure cancer and end all wars, I’m all for it, even if it means I’m personally out of a job.