"The failures are the curriculum"

AI equips the layperson to impersonate an expert, producing output that has all the hallmarks of expert judgement—except for the judgement itself.

I have several family members in academia, so the phenomenon being discussed in this piece was already familiar to me, but Dr. Karamanis paints an evocative picture of the underlying structural problem.

I see a lot of parallels in the tech industry, and I’m trying to quash any nascent tendencies toward this in my own life.

Well-meaning workers are trying their best to be useful as they watch their colleagues fall away around them. They’re inheriting roles and responsibilities they may not be qualified for or equipped to do justice to. Thanks to the same AI that’s bedazzled their leaders, these folks can produce work…but not substantiate it. Unable to understand the model’s output enough to polish it into worthiness, they lean too much on the model’s expertise.

Except models don’t have expertise, they have statistics. Statistics are valuable, but on their own they can only mimic expertise. The missing ingredient is instinct, which is honed through experience. For now, those qualia are beyond the reach of LLMs.

And so those workers find themselves handing in artifacts they’re unable to fully explain, maintain, or improve upon. When corporate cultures confuse volume with impact to justify their AI investments, it creates a hidden liability. That knowledge gap remains buried right up until a crisis makes it suddenly and painfully obvious.

On the other hand, there’s a parallel phenomenon. One I also see plenty of, thankfully.

There’s a way of using these tools to accelerate the building of expertise. Not by skipping the drudgery, but by shortening the time between tries. Lowering the cost of experimentation. And, I suppose, by providing an infinitely patient professor (who’s only sometimes high on mushrooms) that you can appeal to for answers, without fear of meaningful judgement or being made to feel stupid.

You know how some videogames make a big deal out of you dying, while others do their best to get you back in the game as quickly as possible? Which one are you more motivated to keep playing? In this way, AI can make learning faster, and therefore more fun. Nothing teaches faster than fun. This approach does require curiosity, though. You have to be interested in the process too, not just the outcome.

While I’m fairly sure this process doesn’t produce equivalent expertise to previous methods of learning, I do think it’s a valuable improvement on not caring enough to learn at all.

By the way, this seems to be Minas’ blogging debut! Great new blogs are always an exciting find. Go read The machines are fine. I’m worried about us. and then add the website to your feed reader.

Here’s what I highlighted:

The majority of PhD students will leave academia within a few years of finishing. Everyone knows this. The department knows it, the funding body knows it, the supervisor probably knows it too even if nobody says it out loud. Which means that, from the institution's perspective, the question of whether Alice or Bob becomes a better scientist is largely someone else's problem. The department needs papers, because papers justify funding, and funding justifies the department. The student is the means of production. Whether that student walks out the door five years later as an independent thinker or a competent prompt engineer is, institutionally speaking, irrelevant. The incentive structure doesn't just fail to distinguish between Alice and Bob. It has no reason to try.

The models are already powerful enough to produce publishable results under competent supervision. That's not the bottleneck. The bottleneck is the supervision. Stronger models won't eliminate the need for a human who understands the physics; they'll just broaden the range of problems that a supervised agent can tackle. The supervisor still needs to know what the answer should look like, still needs to know which checks to demand, still needs to have the instinct that something is off before they can articulate why. That instinct doesn't come from a subscription. It comes from years of failing at exactly the kind of work that people keep calling grunt work. Making the models smarter doesn't solve the problem. It makes the problem harder to see.

The real threat is a slow, comfortable drift toward not understanding what you're doing. Not a dramatic collapse. Not Skynet. Just a generation of researchers who can produce results but can't produce understanding. Who know what buttons to press but not why those buttons exist. Who can get a paper through peer review but can't sit in a room with a colleague and explain, from the ground up, why the third term in their expansion has the sign that it does.

The failures are the curriculum. The error messages are the syllabus. Every hour you spend confused is an hour you spend building the infrastructure inside your own head that will eventually let you do original work. There is no shortcut through that process that doesn't leave you diminished on the other side.

Schwartz can use Claude to write a paper because Schwartz already knows the physics. His decades of experience are the immune system that catches Claude's hallucinations. A first-year student using the same tool, on the same problem, with the same supervisor giving the same feedback, produces the same output with none of the understanding. The paper looks identical. The scientist doesn't.