Chapter 1.1 β€” What Is a Language Model, Really

Contents

  1. Where the idea comes from: communication as a statistical process
  2. The probabilistic framing: language modeling as next-token prediction
  3. Measuring a language model: entropy, cross-entropy, and perplexity
  4. Why prediction is such a powerful training signal
  5. Interview angle
  6. Self-check questions
  7. Sources

1. Where the idea comes from: communication as a statistical process

Long before anyone trained a neural network on text, the core idea behind language modeling was already on paper: a message can be treated as the output of a random process, and the job of a good model of that process is to predict what comes next as accurately as possible.

That idea comes from Claude Shannon's 1948 paper "A Mathematical Theory of Communication," written to solve an engineering problem that has nothing to do with meaning or grammar: how do you transmit a message over a noisy channel using as few bits as possible? Shannon's insight was that the statistical structure of a message β€” how predictable its next symbol is, given what came before β€” determines how compressible it is. A source that's highly predictable (say, English text, where "q" is almost always followed by "u") needs fewer bits to encode than a source of pure noise, where every symbol is a surprise.

This is worth sitting with, because it's the thread that runs through this entire book. Language modeling did not start as an attempt to build something that "understands" language. It started as an attempt to build something that predicts language well β€” and it turns out that prediction, pushed far enough, forces a model to encode an enormous amount of what we'd casually call understanding, simply because understanding is what makes good prediction possible. Modern LLMs are the extreme end of a line that starts with Shannon treating English as a stochastic source and asking how many bits per letter it really needs.

2. The probabilistic framing: language modeling as next-token prediction

Formally, a language model assigns a probability to a sequence of tokens (words, subwords, or characters β€” we'll get precise about tokenization in a later chapter). Given a sequence of tokens \(x_1, x_2, \ldots, x_n\), a language model estimates:

$$P(x_1, x_2, \ldots, x_n)$$

By the chain rule of probability, this joint probability can always be decomposed into a product of conditional probabilities:

$$P(x_1, \ldots, x_n) = \prod_{i=1}^{n} P(x_i \mid x_1, \ldots, x_{i-1})$$

This decomposition is the single most important equation in this book. It says that modeling an entire sequence reduces to modeling one thing, over and over: given everything that came before, what comes next? Every architecture covered in this book β€” n-gram tables, RNNs, LSTMs, and Transformers β€” is a different engineering answer to the same question: how do we compute \(P(x_i \mid x_1, \ldots, x_{i-1})\) as well as possible, and as efficiently as possible?

This framing is why "language model" and "next-token predictor" are effectively synonyms in this book, and why that's a much stronger notion than it sounds. Predicting the next token well requires tracking syntax (what part of speech is grammatical here), semantics (what word makes sense given the topic), world knowledge (what fact is being referenced), and even reasoning (what conclusion follows from the premises stated so far). A model that predicts the next token in "The capital of the country where the Eiffel Tower is located is" almost has to encode a fact about geography to do it well. Next-token prediction is a deceptively narrow-sounding objective that turns out to have almost unlimited depth once you ask a model to do it at scale, across the full range of human writing.

It's worth being precise about what "the model" is doing mechanically at each step: producing a probability distribution over the entire vocabulary for what token comes next, conditioned on everything so far. Generation is just sampling from that distribution repeatedly, one token at a time, feeding each choice back in as new context β€” a process usually called autoregressive generation. This is true whether the underlying distribution is estimated by counting n-grams in a corpus or by a hundred-billion-parameter Transformer; the framing doesn't change, only the quality and cost of estimating \(P(x_i \mid x_1, \ldots, x_{i-1})\).

3. Measuring a language model: entropy, cross-entropy, and perplexity

If language modeling is about estimating conditional probabilities well, we need a way to measure "well." This is where Shannon's information theory reenters the picture, refined for language modeling specifically by Fred Jelinek and colleagues at IBM in the 1970s, in work on speech recognition.

Shannon's entropy \(H\) measures the average number of bits needed to encode a symbol from a source, given the true distribution of that source. For a language model, we don't know the true distribution of English (nobody does), so we instead measure cross-entropy: how many bits, on average, the model's predicted distribution needs to encode text that actually came from the true distribution. A model that assigns high probability to the tokens that actually occur has low cross-entropy; a model that's frequently surprised by the next token has high cross-entropy.

Perplexity is simply cross-entropy re-expressed on a more interpretable scale: it's 2 raised to the power of the cross-entropy (or \(e\) raised to the power of it, depending on whether you're working in bits or nats). Jelinek and colleagues introduced it explicitly as "a measure of the difficulty of speech recognition tasks" β€” a way to summarize, in one number, how well a model predicts the next word on average. A perplexity of 1 means the model is never surprised β€” it always assigns probability 1 to whatever actually comes next, which only happens for a trivial or memorized sequence. A perplexity of \(V\) (the vocabulary size) is what you'd get from a model that guesses uniformly at random among all possible tokens. A well-trained model on natural text sits somewhere in between, and lower is better: it means, informally, "the model behaves as if it only had to choose between this many roughly-equally-likely options at each step," even though the actual vocabulary might be tens of thousands of tokens.

$$\text{PPL} = 2^{H(p,q)} = 2^{-\frac{1}{N}\sum_{i=1}^{N}\log_2 q(x_i \mid x_{<i})}$$

A related metric worth knowing is bits-per-character (or bits-per-byte, for a tokenizer-agnostic version): instead of measuring surprise per predicted token, it measures surprise per raw character or byte of the underlying text, by dividing the model's total cross-entropy in bits by the number of characters or bytes the predicted tokens actually spell out:

$$\text{BPC} = \frac{H_{\text{bits}}}{N_{\text{characters}}}$$

The reason this matters is that perplexity's numeric value depends on the vocabulary a model's tokenizer happens to use β€” a perplexity of 15 computed under one tokenizer and a perplexity of 15 computed under a different one aren't measuring the same thing, since the two models aren't being surprised about the same-sized chunks of text, and comparing the raw numbers directly is meaningless. Bits-per-character sidesteps that by anchoring the unit of measurement to the text itself rather than to whatever pieces a particular tokenizer happens to split it into, which is exactly why it's the metric of choice when comparing models that use genuinely different vocabularies or tokenization schemes (Chapter 4.1 covers tokenization in depth).

This matters practically because perplexity (and the cross-entropy it's derived from) is the actual training loss for essentially every language model discussed in this book. Every architecture upgrade β€” from n-grams to RNNs to Transformers β€” is, from the perspective of the optimization objective, an attempt to bring cross-entropy down on held-out text. It's worth remembering that this single, simple, task-agnostic loss is what eventually gives rise to models that can hold conversations, write code, and pass professional exams. None of that downstream capability was directly optimized for; it emerged from optimizing prediction accuracy at scale.

It's also worth flagging perplexity's limits early, since they matter for a book about engineering, not just theory: perplexity is a proxy. It correlates well with many kinds of downstream quality, but two models with identical perplexity can behave very differently on tasks like factual accuracy, instruction-following, or safety, because perplexity is computed against a specific evaluation corpus and averages over every token equally, whether that token is a trivial function word or the one word in the sentence that actually mattered. Later chapters on evaluation return to this tension in depth: perplexity is necessary but not sufficient as a measure of whether a model is good.

4. Why prediction is such a powerful training signal

It's worth pausing on why "predict the next token" turned out to be one of the most productive training objectives ever devised in machine learning, because this justifies the entire structure of this book.

First, it's self-supervised: raw text already contains its own supervision signal. You don't need to label anything β€” every sentence ever written is simultaneously a training example and its own ground truth. This means the objective scales with the amount of text available, which turned out to scale extraordinarily well (a claim we'll make precise in the chapter on scaling laws).

Second, it's maximally information-dense per example. Unlike a classification objective with a handful of possible labels, next-token prediction asks the model to distinguish among the entire vocabulary at every single position in every single sequence. A billion-token training corpus doesn't give you a billion labels' worth of signal β€” because of the chain-rule decomposition above, it gives you a prediction problem at every one of those billion positions, each conditioned on a different growing context.

Third, and most importantly for later chapters, the objective is a lossless proxy for "model the data-generating process of human writing." Because human writing encodes facts, reasoning chains, style, dialogue, code, and more, a model that gets good at predicting all of it needs internal machinery that represents all of those things, at least implicitly. This is the conceptual bridge to everything that follows: the rest of this book is the story of how the field kept building better function approximators for \(P(x_i \mid x_1, \ldots, x_{i-1})\) β€” from count tables, to fixed feed-forward networks, to recurrent networks, to attention, to the Transformer β€” each one able to condition on more context, more efficiently, and with better generalization than the last.

5. Interview angle

A senior LLM interview will rarely ask "what is a language model" outright, but it will probe whether you actually understand the framing underneath everything else, usually through questions like:

  • "Why is next-token prediction such an effective pretraining objective?" β€” the expected answer touches self-supervision, information density per token, and the claim that good prediction requires implicit world modeling.
  • "What's the difference between perplexity and accuracy as an evaluation metric, and when would they disagree?" β€” a strong answer explains that perplexity is a smooth, token-averaged proxy, while task accuracy is an outcome-level measure, and that a model can improve one while stalling on the other, especially once a model is well past the regime where perplexity was originally most informative.
  • "Why doesn't lower perplexity always mean a better product experience?" β€” this tests whether you understand perplexity's blind spot: it weighs every token equally, including tokens that don't affect whether an answer is actually useful, correct, or safe.

The through-line interviewers are checking for is whether you see modern LLMs as a scaled-up, better-engineered answer to the same next-token-prediction problem Shannon and Jelinek were already formalizing decades earlier β€” rather than as a categorically new kind of object.

6. Self-check questions

  1. Write out the chain-rule decomposition of \(P(x_1, \ldots, x_n)\) from memory, and explain in one sentence why this decomposition is the reason every language model architecture can be compared on the same footing.
  2. What problem was Shannon actually trying to solve in 1948, and why does a solution to that problem also give you a recipe for language modeling?
  3. Define cross-entropy and perplexity in your own words, and explain what a perplexity of 1 versus a perplexity equal to the vocabulary size would each imply about a model.
  4. Why is next-token prediction described as "self-supervised"? What would a supervised alternative look like, and why doesn't it scale the same way?
  5. Give an example of two model outputs that could have identical perplexity on some corpus but very different quality from a user's perspective. What does this tell you about perplexity as a metric?
  6. Explain, in your own words, why a model trained only to predict the next token might end up implicitly encoding facts, reasoning ability, or style β€” even though none of those were directly part of the training objective.

7. Sources

  • Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27(3) & 27(4). Wiley
  • Jelinek, F., Mercer, R. L., Bahl, L. R., & Baker, J. K. (1977). Perplexity β€” a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America, 62(S1). AIP Publishing

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

Which equation correctly expresses the chain-rule decomposition of $P(x_1, \ldots, x_n)$ used throughout this book?

Explanation: The chain rule factors the joint probability of the whole sequence into a product of next-token conditionals β€” this is why every architecture in the book can be understood as a different way to estimate the same conditional distribution.

What problem was Claude Shannon actually solving in his 1948 paper that gave rise to the language-modeling framing?

Explanation: Shannon's "A Mathematical Theory of Communication" is about efficient, reliable transmission over a noisy channel. The statistical predictability of a source turned out to be exactly what language modeling formalizes decades later.

Who introduced the term "perplexity" as a measure of language model quality, and in what context?

Explanation: Jelinek, Mercer, Bahl, and Baker introduced perplexity explicitly in 1977 as "a measure of the difficulty of speech recognition tasks," building on Shannon's notion of cross-entropy.

Why is next-token prediction described as a "self-supervised" training objective?

Explanation: Every sentence ever written is simultaneously a training example and its own ground truth β€” the next token to predict is just sitting right there in the text, so no manual labeling step is needed.

A language model has a cross-entropy of 4 bits per token on some evaluation text. What is its perplexity?

Explanation: Perplexity = 2^(cross-entropy in bits) = 2^4 = 16.

A model assigns probability 0.0625 to the token that actually occurs next. How many bits of "surprisal" ($-\log_2 p$) does this correspond to?

Explanation: $-\log_2(0.0625) = -\log_2(1/16) = 4$ bits.

A tokenizer's vocabulary has 32,768 tokens. What perplexity would a model get if it guessed uniformly at random among all tokens at every step?

Explanation: A model that guesses uniformly among V tokens gets a perplexity of exactly V β€” here, 32,768.

A model has a cross-entropy of 2 nats per token on held-out text. What is its perplexity, to two decimal places? (Use $e \approx 2.71828$.)

Explanation: Perplexity = e^(cross-entropy in nats) = e^2 β‰ˆ 7.39.

Why is bits-per-character (BPC) preferred over perplexity when comparing models that use different tokenizers?

Explanation: Perplexity's value depends on the vocabulary a tokenizer happens to use, so two models tokenizing text differently aren't being surprised about equal-sized chunks. Dividing total cross-entropy by the number of characters or bytes sidesteps that, making BPC comparable across tokenization schemes.

A model's total cross-entropy over an evaluation text is 8,000 bits, and the predicted tokens spell out 2,000 characters. What is its bits-per-character (BPC)?

Explanation: BPC = total bits / number of characters = 8,000 / 2,000 = 4.