← Back to blog

LLMs were used in every possible way to produce this post.

Super average machines

· 6 min read
LLM Artificial Intelligence

I’ve noticed that when I let an AI design or build something I haven’t thought through first, the result often isn’t what I needed. It picks the easy option, takes shortcuts, or says a task is too big to do properly and suggests a simpler version instead.

Ask a model “what’s the best way to build authentication for a SaaS app?” and you get something like this:

  • Solo founder or small React team shipping an MVP this weekend: use Clerk or Supabase Auth.
  • B2B SaaS selling to mid-market or enterprise: use WorkOS or PropelAuth, so SAML, SCIM and org management don’t become a bottleneck.
  • TypeScript team wanting full ownership and no per-user costs: use Better Auth with your own Postgres.
  • Strict enterprise with multi-language microservices and dedicated DevOps: self-host Keycloak.

It’s a reasonable answer, and a fair summary of what people usually recommend. But unless you’ve tried those options, you don’t know how big the trade-offs between them are, and the answer doesn’t tell you.

Models are useful for sparring with your ideas. They also tend to like all of them, so a “yes, this is a good plan” doesn’t mean much on its own.

I think this follows from how the models work: they give you the average.

Next token predictors

I’m not an expert on how LLMs work, and I have never built one. The writing here is mostly my own mental model, not a technical explanation.

At its core, an LLM predicts the next token based on all the text before it. Then it adds that token to the text and predicts the next one, over and over. It’s a very good and very elegant mechanism, and it may not be that far from how human speech forms either.

Most of the parts aren’t new. Artificial neurons and the perceptron go back to the 1950s, backpropagation became widely known in the 1980s, and neural language models have been around for decades. What changed in 2017 was the transformer architecture, which combined these ideas in a way that scaled. The first GPT models followed a year later. Depending on how you define “language model”, the history goes back much further than many people think.

Many people, including me sometimes, say “it’s just a next token predictor” as a dunk when someone gives LLMs too much credit. I’m not sure that’s entirely fair, because it doesn’t describe the value we actually get from them. The phrase describes the training objective, not what the trained model has learned. To predict the next move in a chess game well, you may need to represent the board, the threats and the plans. To predict the next line of code well, you may need to infer the goal and the constraints.

Picking the most likely answer means picking the average

When the model picks the next token, it picks from a probability distribution. With the temperature at zero, it always takes the most probable token.

If you picture this as a bell curve, the model gives you the middle. Strictly speaking, it’s the most likely continuation and not a statistical mean, and post-training tends to push models even further toward safe, consensus answers, and toward agreeing with you. But the intuition holds: the model gives you the answer that best fits what most of the text it has seen would say.

That’s what we want most of the time. The most likely answer is usually a useful answer. The side effect is that if you ask a model something, you get the average answer.

The average gets worse as the topic gets harder

The knowledge embedded in human language isn’t evenly good. On easy, well-documented topics, the average is great, because almost everyone who wrote about it got it right. The harder and more specialized the topic, the more the average drifts toward what’s common rather than what’s correct or appropriate.

Back to authentication. Ask an agent to “implement authentication” and it takes the path of least resistance: the most typical implementation it has seen. It sets up Clerk, and now you are locked in to an external vendor, forced to adopt their patterns and limitations. That works wonders for a demo, but it won’t necessarily scale, fit your architecture or meet your actual requirements. It becomes a burden later, when you want to do something they don’t support.

Being more specific and saying “I want secure authentication” doesn’t fix the problem either. Security isn’t one dial you turn to maximum. An agent could happily build something “very secure” that’s useless in practice. In theory, it could set the password hashing cost so high that checking a single login takes minutes. Secure, yes. Usable, no.

To get past the average, you need to know your requirements: what’s enough, what isn’t, and what you can live without. Where do you think you’ll be in a few months? The agent can help you answer those questions, and it’s very good at explaining them. But if you don’t know what you want or need, you can’t steer it away from the middle.

Authorship and ownership

The main argument I’m making is that the author has to know what they’re doing. “Build me an app that does X” isn’t enough, and that isn’t specific to LLMs. You couldn’t hand that sentence to a human software team either. You’d get an app. Cool. What does it do?

Doing something useful and novel usually doesn’t mean inventing completely new things. It means combining things we already know in new ways: incremental innovation. Most science is like that too. Completely novel discoveries are rare. Most progress is small: someone takes existing ideas, methods or findings and puts them together in a way nobody had tried yet. Even the LLM itself is an example of this. It was built mostly from pieces that had existed for decades, combined in a new way.

Each building block can be perfectly average, and LLMs are great at producing average building blocks. The novelty is in knowing which blocks to combine, where to look and why, and that part has to come from the person.

Humans aren’t only language

Even if language production works something like next-token prediction, being human isn’t only language. We attach feelings, experiences and stakes to things. Our brains do a lot more than what the language centers do. When a person decides what “enough” security means for their app, that decision draws on context, consequences and things they care about that never made it into any text. That’s exactly the part a model working from the average of written language can’t supply on its own.

That’s also why I don’t think everyone in the IT industry will lose their job. People who mechanically do what they are told are likely to lose theirs. People who can think for themselves and steer things in a good direction will have more work than ever.

Knowing what to ask

I kind of like the term “super average machines”. LLMs are like that. They give you the most likely answer. Most of the time that’s exactly what you want, and it’s the reason they’re so useful. But the most likely answer is the average answer, and the average gets worse the harder your problem is, and the vaguer your question is.

That’s what the authentication answer at the start shows: a reasonable list, but choosing from it is still your job. The quality of what you get out depends heavily on what you bring in: knowing the trade-offs, knowing what’s enough and knowing which average pieces to combine into something new. The models can help you learn those things.