Part 2 of 4 in The Brilliant Stranger, a field guide to LLMs, explained like you run a business.
Last time: the model is an expert behind a door, and it meets you for the first time every conversation. This time, the question that comes right after.
“Shouldn’t it get smarter the more we use it?” It’s a fair assumption. A new hire who answers the same question a hundred times gets faster and sharper. Even if the expert can’t remember you, surely all that use adds up to something.
It doesn’t. The model you’re talking to stopped learning before you ever met it. To see why, you have to look at where its knowledge actually lives.
The wrong picture: a filing cabinet
Ask most people where an LLM’s knowledge lives and they’ll describe something like a filing cabinet: a drawer for Python, a folder for Spanish grammar, a tab for the French Revolution. Ask a question, and the model reads back the right folder. Under that picture, learning is easy. Every correction is a new page in a folder, and the cabinet gets fuller the more you use it. If that were true, most of what makes AI at work frustrating would not be.
The better picture: a guitar tuned at the factory
A model’s knowledge lives in its weights, a very large set of numbers, hundreds of billions of them. Picture an instrument with that many tuning pegs, where every song depends on the whole instrument being tuned a particular way.
There is no Python peg, no Spanish grammar peg. Those capabilities live in the overall pattern, the way a chord lives in the relationship between strings, not in any single one. Training is the process of finding a tuning where the instrument plays every song in the data reasonably well at once. That takes months of computing and an enormous budget, and it happens before the model ships.
Then the pegs are locked. Playing the guitar doesn’t retune it, and neither does talking to it. Every conversation starts from exactly the same tuning, and when the conversation ends, nothing about the instrument has changed. A model that has answered the same question ten thousand times is no better at it the ten thousand and first.
So what feels like learning?
Last time, the assistant outside the door kept notes and slid them back in. That’s the only learning happening. In guitar terms, the musician hasn’t learned your song. Someone put the right chart on the music stand before you asked.
That changes what “making it smarter” means. You don’t improve the model by using it. You improve what it gets handed: which chart, how current, how clearly written, and who decides. That’s the rest of this series.
Can’t we just retune it ourselves?
You can. That’s what fine-tuning is: training the model a little more on your price book, your playbook, your documents. But you’re turning pegs so one new song sounds right, and every other song was relying on those pegs staying put. The model gets worse at things it used to handle fine, sometimes things nobody thought to test. Researchers call this catastrophic forgetting.
It also doesn’t solve the original problem. Your pricing changes next quarter and the tuning is stale, and when a better model comes out, the work starts over. Fine-tuning earns its cost when you need the model to reliably hold a format, tone, or kind of task that instructions alone can’t. For facts, policies, and anything that changes, it’s the wrong tool.
What this means for your business
Stop expecting use to make it smarter. On its own, it won’t. A correction that isn’t written down somewhere the system can hand back is gone when the conversation ends.
Put your knowledge next to the model, not inside it. The filing cabinet is real. It just sits beside the instrument. Keep it current, and the answers stay current.
Plan for the model to change. Your documents, instructions, and tests carry over to next year’s model. Retuned weights don’t.
Don’t retune the guitar. Hand it the right chart.
Next time: the chart itself. Most companies hand the AI their documents and get worse answers than they expected. It’s fixable.
Wondering why your AI isn’t getting any smarter? Reach out.