Reading is a sequence. Word after word, idea after idea, something takes shape. For people, it’s meaning. For machines—at least the kind behind tools like ChatGPT—it’s prediction.
Large language models (LLMs) are a type of artificial intelligence trained to generate text. They don’t understand language the way we do. They don’t think or reflect. But they’re trained to spot patterns in how people talk, write, and structure thoughts. And they do this not by understanding the meaning of each word—but by calculating which word is most likely to come next. They build responses not from meaning, but from structure.
They build responses not from understanding, but from structure.
Not from intention, but from attention.
Here’s how it works.
Let’s say the sentence reads:
The cat sat on the…
The model assigns a set of probabilities:
- mat → 60%
- floor → 20%
- roof → 5%
- table → 5%
Rather than always picking the top word, the model samples from the distribution. That means mat is more likely, but floor or roof still have a chance. This keeps the output flexible, avoids stiffness, and better reflects the natural rhythm of language.
What makes this possible is a system called a Transformer, and at the heart of that system is something called attention.
Pay attention
Attention mechanisms allow the model to weigh all the words in a sentence—not just the last one—crafting its focus based on structure, tone, and context.
Consider:
“The bank was…”
A basic model might guess the next word with this level of likelihood:
- open → 50%
- closed → 30%
- muddy → 5%
But now add more context:
“So frustrating! The bank was…”
Suddenly, the prediction shifts:
- closed → 60%
- open → 10%
- muddy → 20%
The model has reweighted its focus. “So frustrating” matters. It’s not just responding—it’s recalculating what’s relevant to the meaning of the sentence.
Behind the Scenes: Vectors and Embeddings
To do that, it converts each word into something called a word embedding—a mathematical representation of the word’s meaning based on how it appears across countless examples of language. You can think of it as placing each word in a multi-dimensional space, where words with similar uses and associations are grouped closely together. Each embedding is a type of vector—a set of numbers that places the word in a multi-dimensional space based on how it’s used.
Words like river and stream may live near each other because they’re used in similar ways. But imagine the space of language as layered: piano and violin might be close in a musical dimension, but distant in form. Shark and lawyer—biologically unrelated—might still align on a vector of aggression or intensity. Even princess and daisy could drift together in a cluster shaped by softness, nostalgia, or gender coding.
The model maps relationships among words by how words co-occur. Similarity becomes a matter of perspective: a word might be near in mood, but far in meaning. Embedding captures that layered closeness—a sense of how words relate, not by definition, but by use.
In most modern large language models—including ChatGPT—each word is represented by three vectors:
- Query – what this word is looking for
- Key – what other words offer
- Value – the content to possibly pass forward
The model compares each word’s Query to every other word’s Key using a mathematical operation called a dot product, which measures how aligned two vectors are. You can think of it like angling searchlights—if the direction of one light (the Query) closely overlaps with another (the Key), it suggests the second word offers the kind of information the current word is searching for. These alignment scores reflect how useful or relevant one word is in predicting another. In essence, the model is computing how well each Key meets the needs of the current Query.
But relevance alone isn’t enough. These scores are then passed through a function called softmax, which does two things: it scales the numbers down to keep any one score from overpowering the others, and it transforms them into a probability distribution that adds up to 1. This lets the model share its attention across multiple words—perhaps giving 70% of its focus to “so frustrating,” 20% to “bank,” and 10% to “was,” depending on which words feel most informative.
Finally, the model uses these attention weights to blend the Value vectors—the raw information each word offers—into a single context-aware signal. That signal becomes the lens through which the model predicts the next word. It’s not simply remembering—it’s composing, drawing forward meaning based on what the sentence has revealed so far.
Why It Matters
This is why models like ChatGPT can manage long sentences, track pronouns, and maintain tone.
It’s not because they know the rules. It’s because they weigh the sentence’s structure with attention, step by step.
Still—they aren’t human. They don’t reflect or feel. But they register patterns and adjust as a sentence unfolds.
That’s what makes it powerful—and sometimes uncanny.
The Deeper Thread
Reading skill is closely tied to sequence learning. We don’t just absorb facts—we follow shapes, trace threads. And machines, in their own way, are learning to do the same.
If we want to understand how language models work, we have to understand how they handle sequences—how they learn from them, how they move through them, how they reshape what comes next.
Every word shapes what comes next and reshapes what came before. Every word reshapes the space around it.
Not just for us. But now for the systems we build.