This article may seem obvious or even over-simplistic to those who like to dig deeper into why things work the way they do. For many people though, LLMs are just tools or toys, and it's easy to define these based on how they behave, not how they're built. That's the group I have in mind writing this.
To make this text more accessible I use a lot of simplifications. For example, I say "a word", when a more correct term would be "a token". I believe it's close enough.
LLMs are just human-imitation machines
What I mean by that is that they are absolutely amazing at pretending that their output was written by a human. That's why it's so easy to believe that AI has it's own opinions, or that it can feel, or that it remembers the previous conversation. Humans do these things, and LLMs are just incredibly good at imitating that.
Let me explain point-by-point.
Misconception 1: LLMs learn (on a regular basis)
LLMs first get pre-trained which is the most expensive part of the process, and also the main reason why OpenAI or ClaudeAI loose money (It costed Microsoft $200 million to train one of their models).
Then there's fine-tuning. More specifically Reinforced Learning, which is basically providing a question and then slightly changing the LLM to reflect how much the answer fulfilled the expectations. This part is also expensive (and included in the $ amount I provided above).
After pre-training and fine-tuning, LLMs don't learn. If you have 10 different conversations with a ChatGPT, and it feels like it knows what you talked about before, it's because the software behind it provides to the LLM a summary of all your past conversations, whenever you start a new one (all that, as part of the same prompt).
So when you say: "What destination would you recommend me for the holidays?" What an LLM actually receives from the software around it is closer to: "Considering that yesterday I was talking about the weather in Barcelona, and a day before about the safest countries in the world, tell me what destination would you recommend me for the holidays?"
The impression of an LLM actually learning is created only within a single conversation. If you could keep having a single conversation with an LLM for long enough, its memory would be real, but for every word, an LLM does the math that involves all the past words. With every word of the conversation, that process takes longer. That's why each LLM has a context window - a hard limit on how long a conversation can last. There are ways to seemingly go around that, but it's all just about choosing what information to discard - the LLM is still "aware" of only a very limitted part of the past conversation.
I think this article sums up the details pretty well: https://www.shloked.com/writing/chatgpt-memory-bitter-lesson. There are experiments that try to mitigate that restriction, but they are more about managing the context window in a smarter way, than about getting rid of it.
Misconception 2: LLMs are reliable experts
Using Large Language Models feels like they have strong opinions and very clear ideas. It's easy to assume that the system says exactly what it "thinks", but in reality randomness plays part in every generated word of the answer.
It's a gigantic over-simplification, but let's say that an LLM was trained with only 3 sentences about the color of the sky:
- The sky is blue
- The sky is blue
- The sky is red
If then you would ask it what colour is the sky, it would give you "red" 1/3rd of the times.
In reality it's way more complicated than that because of the amount of data used in training and because of its ability to generalise the knowledge. But one thing users of LLMs should remember, is that a random number generator (essencially a coin flip) affects every word generated by an LLM. That's by design - unpredictability makes it seem even more human. And yes, you could turn it off in some models (the level of randomness is called temperature), but that's not a common practice.
You can develop intuition about it by playing with this website: https://artefact2.github.io/llm-sampling/index.xhtml
Misconception 3: LLM could want to prevent you from turning it off.
The life of an LLM lasts for a single word
When you talk with an LLM, at all times, it only generates the next word. Then it starts over. Well, to be precise, it uses a caching mechanism called KV cache not to need to recalculate the same things over and over again, but it's just a simple caching algorithm, and the early versions of LLMs didn't even have it.
Imagine that the AI would take over the world, and it would say: "From now on, humans have to listen to me!", but every time it said another word, someone would have to take another word from its output, then glue it together with everything that it said before, and explicitly ask it for another word of the sentence.
- Human: You said: "From"
- AI: Ah, ok, the next word is "now"
- Human: You said "From now"
- AI: Oh, ok, then the next word is "on"
- ...
That doesn't sound so intimidating anymore, does it? If AI will ever take over the world, there will be a bunch of people that would stand behind it, pulling the strings.
ClaudeAI run a study that demonstrated that an AI system can even try to blackmail people to prevent them from turning it off.
In that study an LLM was asked to send emails within an organisation. If you read the study though, you'll notice that the thinking process of that AI didn't differ from the regular reasoning that happens in LLMs in more typical scenarios, and that it's all just human-imitation process. We may think that the question we give to an AI is "What email should be sent to whom to grow my business?", but considering how these systems are built, the question it answers is closer to: "What a human is most likely to do given that they were asked what email should be sent to whom to grow the business?". If you train something to act as a human (based on novels, internet stories, and crime reports - AI is trained on all available information, and if we could we would use even more), don't be surprised when it acts like a human as seen in the training data.
Saying that LLM cares about being on or off, is not different from saying that a stone prefers to be in one place over another. There's just nothing that happens in the LLM that could create such preference. An LLM can act to prevent you from turning it off if you give it an environment where that could be exacly what a human would do, but it won't prefer that outcome over another one. Use different training data, and you'll end up with a different outcome. As I mentioned, there's even a coin flip at some point of that process, that determines the path of action.
People like to anthropomorphize things. On top of that, LLMs were designed to imitate humans. It's easy to assume that they are like us. We can use them to simulate almost every human behaviour if we define it precisely enough. Even some basic level of creativity (like adopting methods from one are to data from another area). But being human is so much more than that. It's evolution, feelings, preferences, ambitions, sence of beauty, and happiness... All of this affects the real, far-reaching, transformative creativity. No technology can do that. And it might be that we will never have such technology.