Chapter 1.3 β€” Neural Language Models Before RNNs

Contents

  1. The idea that broke the n-gram ceiling: distributed representations
  2. Bengio's neural probabilistic language model
  3. Word2Vec: representations as the product, not the byproduct
  4. GloVe: counting your way to the same place
  5. What these representations do and don't solve
  6. Interview angle
  7. Self-check questions
  8. Sources

1. The idea that broke the n-gram ceiling: distributed representations

Chapter 1.2 ended on the deepest limitation of count-based language models: they treat every word as an atomic, unrelated symbol. Learning that "the cat sat on the mat" is a plausible sentence tells an n-gram table nothing about "the dog sat on the rug," even though a person immediately recognizes these as structurally interchangeable. There is no notion of similarity between "cat" and "dog" in a system built purely out of counting distinct strings.

The fix is to stop representing words as distinct symbols and start representing them as points in a continuous vector space, where similar words end up near each other. This is the idea of a distributed representation: instead of one word being one arbitrary index into a giant table, a word becomes a vector of, say, a few hundred numbers, and those numbers are learned so that words used in similar ways end up with similar vectors. Once words live in a continuous space, a model can generalize across them the way arithmetic generalizes across numbers: learning something useful about the region of space near "cat" tells you something about "dog," because they land close together.

This single idea β€” replacing symbolic identity with a learned continuous representation β€” is arguably the most important conceptual shift in the entire history of language modeling, more foundational than any specific architecture. Everything from this chapter through the largest modern Transformers is a story about what to do once words (or subwords, or pixels, or anything else) are represented this way.

2. Bengio's neural probabilistic language model

The paper that first made this idea concrete and end-to-end trainable for language modeling was Yoshua Bengio and colleagues' 2003 "A Neural Probabilistic Language Model." The architecture is, by modern standards, almost quaint: a feedforward neural network takes the previous \(n-1\) words as input, looks each one up in a shared table of learned vectors (the embedding table), concatenates those vectors, passes the result through one or two hidden layers, and outputs a probability distribution over the vocabulary for the next word β€” trained, just like every language model in this book, to minimize cross-entropy on the actual next word.

Three things about this design are worth calling out because they persist, in one form or another, through every chapter that follows. First, the embedding table is learned jointly with the rest of the network, driven purely by the language modeling objective β€” nobody hand-specifies what makes two words similar; it falls out of training on the prediction task itself. Second, the embedding table is shared: the vector learned for "cat" is the same regardless of where in the input window it appears, so anything learned about "cat" in one context transfers to every other context where it shows up. Third, and most importantly for the sparsity problem from Chapter 1.2, this model can assign a sensible, non-zero probability to a context it never saw during training, as long as the individual words in that context are similar (in the learned vector space) to words it saw in other contexts. This is exactly the generalization that count-based n-grams structurally could not provide.

The model still used a fixed-size context window, exactly like an n-gram model β€” it is still fundamentally a Markov model, just estimated by a neural network instead of a count table. Solving the fixed-window limitation itself would take recurrent networks, the subject of the next chapter. What Bengio's paper solved was the other half of the problem: generalization across similar words within whatever window you do use.

3. Word2Vec: representations as the product, not the byproduct

A decade later, Tomas Mikolov and colleagues at Google published two papers, in 2013, that took the embedding idea from Bengio's model and made it the entire point, rather than a side effect of training a full language model. This is Word2Vec, and it comes in two closely related training schemes: continuous bag-of-words (CBOW), which predicts a target word from its surrounding context words, and skip-gram, which predicts the surrounding context words from a target word. Crucially, Word2Vec drops the deep hidden layers and heavy machinery of a full language model β€” it's a shallow, extremely fast-to-train objective whose entire purpose is to produce good word vectors as the output, not to be a good next-word predictor in its own right.

This shift in framing mattered enormously in practice. Word2Vec could be trained on corpora far larger than was practical for a full neural language model at the time, and the resulting vectors turned out to have a strikingly useful property: simple vector arithmetic captured real semantic and syntactic relationships. The famous example is that the vector for "king" minus "man" plus "woman" lands close to the vector for "queen" β€” meaning relationships between words (here, roughly, "royalty" and "gender") are encoded as consistent directions in the vector space, not just proximity between individual points. Word2Vec's efficient estimation paper and its companion paper on distributed representations (introducing negative sampling, a training trick that made skip-gram practical at scale) together established that "learn good word vectors first, use them everywhere downstream" was a viable and enormously useful strategy on its own, independent of any single language modeling task.

It's worth being precise about what negative sampling actually does, since it's often cited without explanation. Computing skip-gram's real softmax over the full vocabulary β€” tens or hundreds of thousands of words β€” at every training step would be prohibitively expensive, because the softmax's normalizing sum requires touching every vocabulary entry just to predict one context word:

$$P(w_O \mid w_I) = \frac{\exp(v'^{\top}_{w_O} v_{w_I})}{\sum_{w=1}^{|V|} \exp(v'^{\top}_{w} v_{w_I})}$$

Negative sampling sidesteps this by turning the problem into a much cheaper one: instead of predicting a full probability distribution over every possible context word, train a binary classifier that distinguishes the one true (target, context) pair from a handful of randomly sampled (target, random word) pairs β€” typically 5 to 20 negatives per positive example β€” which touches only a small, fixed number of vocabulary rows per step regardless of how large the vocabulary is. The objective for a single (target, context) pair is then

$$\log \sigma(v'^{\top}_{w_O} v_{w_I}) + \sum_{i=1}^{k} \mathbb{E}_{w_i \sim P_n(w)}\big[\log \sigma(-v'^{\top}_{w_i} v_{w_I})\big]$$ Hierarchical softmax, an earlier and less widely adopted answer to the same bottleneck, instead arranges the vocabulary as a binary tree and turns predicting one specific word into a sequence of binary decisions down the tree, which brings the cost from linear in vocabulary size down to logarithmic. Both tricks attack the identical problem β€” an expensive normalization over an enormous vocabulary β€” from different directions: negative sampling by approximating the normalization away entirely, hierarchical softmax by restructuring the computation so the exact answer no longer requires touching every word.

4. GloVe: counting your way to the same place

Around the same time, Jeffrey Pennington, Richard Socher, and Christopher Manning at Stanford published GloVe (Global Vectors), which arrives at similarly useful word vectors from a different starting point: instead of a local prediction task (predict a word from its neighbors), GloVe explicitly factorizes a global word co-occurrence matrix β€” essentially, for every pair of words, how often they appear near each other across the entire corpus β€” into vectors, such that the dot product of two word vectors approximates the log of their co-occurrence statistic:

$$w_i^{\top} \tilde{w}_j + b_i + \tilde{b}_j \approx \log X_{ij}$$

The practical outcome β€” vectors with the same kind of useful arithmetic structure Word2Vec produced β€” is similar, but the underlying philosophy is worth understanding because it recurs constantly in machine learning: Word2Vec is a predictive, local, online approach (learn from one context window at a time), while GloVe is a count-based, global, batch approach (aggregate statistics over the whole corpus first, then factorize). That two quite different training procedures converge on vectors with similar useful properties is itself informative: it suggests the word-vector-with-linear-structure result is not an artifact of one specific training trick, but reflects something real about the statistical structure of how words co-occur in natural language.

One more variant is worth naming because of where it points forward to: fastText, developed by Bojanowski and colleagues at Facebook AI Research, keeps Word2Vec's skip-gram training objective essentially unchanged but represents each word as the sum of embeddings for its constituent character n-grams rather than a single opaque vector per whole word. A word the model never saw during training β€” a typo, a rare inflection, a brand-new coinage β€” still gets a usable vector, assembled from whatever character n-grams it shares with words the model did see, instead of falling back to an out-of-vocabulary placeholder the way whole-word Word2Vec and GloVe both have to. That move β€” represent a word as a composition of smaller, reusable subword pieces rather than an indivisible unit β€” is exactly the idea Chapter 4.1's subword tokenization (byte-pair encoding and friends) takes even further, applying it not just to embeddings but to the vocabulary itself.

5. What these representations do and don't solve

It's worth being precise about what pretrained word embeddings actually fixed and what they left untouched, because both halves matter for understanding why RNNs and then Transformers were still necessary.

What they fixed: the generalization gap from Chapter 1.2. A model equipped with Word2Vec or GloVe vectors (or vectors learned jointly, as in Bengio's model) can transfer statistical strength between similar words, dramatically softening the sparsity problem, and can be pretrained on enormous unlabeled corpora and then reused across many downstream tasks β€” an early, narrower version of the pretrain-then-adapt paradigm that dominates the rest of this book.

What they didn't fix: context length and word-order sensitivity beyond a fixed window, and β€” importantly β€” a single fixed vector per word regardless of context. "Bank" gets exactly one Word2Vec vector, whether it's being used to mean a riverbank or a financial institution; the representation has no way to adjust based on the sentence it's actually appearing in. This limitation β€” static, context-independent embeddings β€” is precisely what contextual representations (built by RNNs, and later by Transformers, both covered in upcoming chapters) were built to solve. The word vector idea introduced in this chapter doesn't go away in later architectures; it becomes the input layer that every subsequent architecture builds on top of, refining a word's fixed vector into a context-dependent one as it flows through the network.

6. Interview angle

This chapter tends to surface in interviews as a check on whether you understand embeddings as a concept, independent of any specific downstream architecture:

  • "What's the difference between Word2Vec and GloVe, conceptually?" β€” the expected answer contrasts local/predictive versus global/count-based training, not just "they're both word embeddings."
  • "Why does 'king - man + woman β‰ˆ queen' work?" β€” a good answer explains that relationships are encoded as consistent vector directions because of how the training objective ties co-occurrence statistics to geometry, not as a mysterious emergent trick.
  • "What's the fundamental limitation of Word2Vec/GloVe embeddings that later architectures had to solve?" β€” the expected answer is context-independence: one fixed vector per word regardless of usage, which motivates contextual embeddings.
  • "Why did Bengio's 2003 model matter if Word2Vec/GloVe came a decade later and are more commonly used?" β€” tests whether you understand lineage: Bengio's paper established that embeddings could be learned jointly with a neural LM objective at all; Word2Vec and GloVe are refinements of how to get good embeddings efficiently, not a different idea.

7. Self-check questions

  1. Explain, in your own words, what a "distributed representation" is and why it lets a model generalize across similar words in a way an n-gram table cannot.
  2. Describe the architecture of Bengio's 2003 neural probabilistic language model, and explain which specific problem from Chapter 1.2 it solves and which one it leaves unsolved.
  3. What is the difference between the CBOW and skip-gram training objectives in Word2Vec?
  4. Contrast GloVe's training approach with Word2Vec's. Why might it be significant that both arrive at vectors with similar useful properties?
  5. What does it mean for a word embedding to be "context-independent," and why is that a real limitation? Give an example word whose meaning changes with context to illustrate.
  6. Trace the throughline from Chapter 1.2's sparsity problem to this chapter's distributed representations to the next chapter's need for context-dependent representations. What specific gap does each step close, and what gap does it leave open?

8. Sources

  • Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3, 1137–1155. JMLR
  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781
  • Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. arXiv:1310.4546
  • Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. EMNLP 2014. ACL Anthology D14-1162
  • Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching Word Vectors with Subword Information (fastText). TACL. arXiv:1607.04606

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

What problem does representing words as vectors in a continuous space solve, that a pure count-based n-gram table structurally cannot?

Explanation: Once words are points in a continuous space, learning something useful near "cat" tells you something about "dog" too, because they land close together β€” count tables have no such notion of similarity.

In Word2Vec, what does the skip-gram training objective predict?

Explanation: Skip-gram goes from target word to context; CBOW (continuous bag-of-words) goes the other way, predicting the target word from its surrounding context.

How does GloVe's training approach differ philosophically from Word2Vec's?

Explanation: Word2Vec is predictive/local/online (one context window at a time); GloVe is count-based/global/batch (factorize a corpus-wide co-occurrence matrix). Both converge on vectors with similar useful linear structure.

What key limitation do Word2Vec and GloVe embeddings share, which motivated the move to contextual representations (RNNs, then Transformers)?

Explanation: Static embeddings assign one vector per word type, with no way to adjust based on the sentence it appears in. Fixing this β€” making the vector depend on context β€” is exactly what RNNs and Transformers were built to do.

Bengio's 2003 model uses a 100-dimensional embedding for each word and a context window of the previous 4 words. What is the size of the concatenated input vector fed into the hidden layer?

Explanation: 4 words Γ— 100 dimensions each, concatenated = 400.

A vocabulary has 10,000 words, and each word is represented by a 50-dimensional embedding vector. How many total parameters are in the embedding table alone (ignoring any other layers)?

Explanation: 10,000 words Γ— 50 dimensions = 500,000 parameters.

Using simplified 2D toy vectors $\text{king} = (5, 3)$, $\text{man} = (4, 1)$, $\text{woman} = (2, 4)$, compute $\text{king} - \text{man} + \text{woman}$ (which should land near "queen"). Give the sum of the resulting vector's two components.

Explanation: king βˆ’ man + woman = (5βˆ’4+2, 3βˆ’1+4) = (3, 6). Sum of components = 3 + 6 = 9.

Two word vectors are $\text{cat} = (1, 2)$ and $\text{dog} = (1.5, 2.2)$. What is the Euclidean distance between them? (Round to 2 decimal places.)

Explanation: $\sqrt{(1.5-1)^2 + (2.2-2)^2} = \sqrt{0.25 + 0.04} = \sqrt{0.29} \approx 0.54$ β€” a small distance, reflecting that the two words are semantically close.

What does negative sampling do to make skip-gram training practical at scale?

Explanation: Instead of normalizing over the entire vocabulary, negative sampling trains a binary classifier to distinguish one true (target, context) pair from a few randomly sampled negative pairs, touching only a small fixed number of vocabulary rows per step.

How does fastText handle a word it never saw during training, unlike whole-word Word2Vec or GloVe?

Explanation: fastText represents each word as the sum of embeddings for its constituent character n-grams rather than a single opaque per-word vector, so an unseen word still gets a usable vector assembled from shared subword pieces.