Chapter 1.4 β€” Recurrent Networks as Language Models

Contents

  1. Fixing the fixed window: the recurrence idea
  2. Vanilla RNNs and why they're hard to train
  3. LSTMs: gating as a fix for vanishing gradients
  4. GRUs: a simpler gate, similar results
  5. Sequence-to-sequence models and the encoder-decoder bottleneck
  6. Interview angle
  7. Self-check questions
  8. Sources

1. Fixing the fixed window: the recurrence idea

Every model in the previous two chapters shares one structural limitation: they condition on a fixed-size window of previous tokens, whether that window is the last \(n-1\) words in an n-gram table or the fixed input slots in Bengio's feedforward network. Making the window bigger helps a little, but it doesn't scale β€” you still have to pick a hard cutoff, and anything before that cutoff is invisible to the model, no matter how relevant it might be.

Recurrent neural networks (RNNs) remove the fixed window by introducing a hidden state that gets updated at every time step and carried forward indefinitely. At each position \(i\), the model reads the current input token and combines it with the hidden state summarizing everything before it, producing a new hidden state:

$$h_i = f(h_{i-1}, x_i)$$

Critically, \(h_i\) is a fixed-size vector, but it's a function of the entire sequence so far, not just the last few tokens β€” in principle, information from position 1 can still influence the hidden state at position 1000. This is the key conceptual leap of this chapter: instead of truncating context (n-grams) or processing a fixed window all at once (Bengio's model), an RNN compresses an arbitrarily long history into a fixed-size summary vector, updated incrementally, one token at a time. Predicting the next token, exactly as in Chapter 1.1, becomes a function of this hidden state: \(P(x_{i+1} \mid x_1, \ldots, x_i) \approx g(h_i)\).

2. Vanilla RNNs and why they're hard to train

The simplest version of \(f\) above (sometimes called an Elman network, after Jeffrey Elman's 1990 formulation of the idea for sequence modeling) is a single layer that combines the previous hidden state and current input through a weight matrix and a nonlinearity. This is elegant and, in principle, exactly what's needed. In practice, it runs into a serious training problem.

Training an RNN means backpropagating the error at some late time step all the way back through every previous time step β€” a procedure called backpropagation through time. Because the same weight matrix is applied repeatedly, once per time step, the gradient at step \(i\) depends on a product of many copies of that matrix (and the derivatives of the nonlinearity applied at each step):

$$\frac{\partial h_t}{\partial h_k} = \prod_{j=k+1}^{t} \frac{\partial h_j}{\partial h_{j-1}} = \prod_{j=k+1}^{t} W^\top \operatorname{diag}\bigl(f'(h_{j-1})\bigr)$$

If the relevant values are consistently less than one, that product shrinks toward zero exponentially fast as you go further back β€” the vanishing gradient problem. If they're consistently greater than one, the product grows exponentially instead β€” exploding gradients. In practice, vanilla RNNs suffer overwhelmingly from the vanishing case: gradients from errors many steps in the future essentially fail to reach and update the parts of the network responsible for information many steps in the past.

The practical consequence is exactly the property the recurrence was supposed to provide: long-range dependency modeling. A vanilla RNN can, in theory, remember something from 100 steps ago, but in practice, the vanishing gradient problem means it's very hard to train it to actually do so, because the training signal that would teach it to preserve that information never reliably arrives. Exploding gradients are the more tractable of the two problems β€” they can usually be controlled with gradient clipping, capping the gradient's norm during training β€” but vanishing gradients required a more fundamental architectural fix.

There's also a practical compute problem sitting alongside the gradient problem: backpropagating all the way through a long sequence is itself expensive in both time and memory, since it means keeping every intermediate activation from every time step around until the backward pass reaches it. Training almost always uses truncated BPTT instead of the full version: run the recurrence forward as normal across the whole sequence, but only backpropagate the gradient through a fixed, limited window of the most recent time steps rather than all the way back to the sequence's start. This trades away some long-range gradient signal β€” errors more than the truncation window away from where they originated never get a chance to update the weights directly β€” in exchange for a training step that fits in memory and finishes in reasonable time, which is a second, independent reason (beyond vanishing gradients themselves) that vanilla RNNs struggle to learn dependencies spanning very long ranges.

One more limitation worth naming separately from the gradient problem: a vanilla RNN processing a sequence left-to-right only ever has access to past context at a given position, never future context. Bidirectional RNNs address this by running two separate RNNs over the same sequence β€” one left-to-right, one right-to-left β€” and concatenating their hidden states at each position, so every position's representation incorporates context from both directions at once. This only makes sense when the entire input is already available before processing starts, which is exactly the situation Chapter 1.5's sequence-to-sequence encoder is in: its job is to summarize a complete, already-fully-known input sequence, so there's no reason to withhold future context from it the way a genuine left-to-right generator producing output one step at a time has to.

3. LSTMs: gating as a fix for vanishing gradients

Long Short-Term Memory networks (LSTMs), introduced by Sepp Hochreiter and JΓΌrgen Schmidhuber in 1997, solve the vanishing gradient problem with an architectural change specifically designed to let gradients flow backward through many time steps largely unimpeded.

The key idea is to maintain a separate cell state, alongside the hidden state, that is updated primarily through addition rather than repeated multiplication by a weight matrix. Three learned gates control this cell state at each step: a forget gate decides what fraction of the existing cell state to keep, an input gate decides what new information to write in, and an output gate decides what part of the (updated) cell state to expose as the hidden state used for prediction. Each gate is itself a small neural network (a sigmoid-activated linear layer) that outputs values between 0 and 1, acting as a soft, learned switch.

Why does this fix vanishing gradients? Because the cell state's update is dominated by an additive term β€” roughly, "keep some of the old cell state, plus add some new information" β€” rather than a repeated matrix multiplication at every step:

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t, \qquad f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$$

Gradients flowing backward through this additive path don't get squashed by the same repeated-multiplication effect that kills vanilla RNN gradients. This lets an LSTM learn to preserve information across long stretches of a sequence when the forget gate learns to stay open (close to 1) for the relevant content, something a vanilla RNN structurally struggles to do regardless of how long you train it.

This gating mechanism is worth understanding at this level of detail because "let the network learn what to keep and what to discard, via a soft additive gate" is an idea that recurs, in different guises, throughout the rest of the book β€” including, arguably, in the residual connections that make very deep Transformers trainable at all.

4. GRUs: a simpler gate, similar results

Gated Recurrent Units (GRUs), introduced by Kyunghyun Cho and colleagues in 2014 (in the same paper that introduced the RNN encoder-decoder framework discussed below) and further popularized by Junyoung Chung and colleagues' empirical comparison later that year, simplify the LSTM's three-gate design into two gates β€” an update gate and a reset gate β€” and drop the separate cell state, folding everything into a single hidden state:

$$z_t = \sigma(W_z[h_{t-1}, x_t]), \quad r_t = \sigma(W_r[h_{t-1}, x_t]), \quad h_t = (1 - z_t)\odot h_{t-1} + z_t \odot \tilde h_t$$

The practical motivation was efficiency: fewer parameters and fewer operations per time step, at roughly comparable performance to LSTMs on many tasks. Which of the two performs better varies by task and dataset, and neither dominates the other universally β€” the more important point for this book is that both are answers to the same underlying problem (vanishing gradients in vanilla RNNs), solved via the same underlying mechanism (learned additive gating), packaged with different levels of complexity.

5. Sequence-to-sequence models and the encoder-decoder bottleneck

So far, this chapter has discussed RNNs as pure language models: process tokens one at a time, predict the next one. But many real tasks β€” translation being the motivating example throughout this part of the book's history β€” require mapping one whole sequence to a different whole sequence, of possibly different length, in a different vocabulary.

Ilya Sutskever, Oriol Vinyals, and Quoc Le's 2014 sequence-to-sequence (seq2seq) paper solved this with an elegant two-network design: an encoder RNN reads the entire input sequence (say, a sentence in French) one token at a time and produces a final hidden state summarizing the whole sentence; that single fixed-size vector is then handed to a separate decoder RNN, which generates the output sequence (the English translation) one token at a time, conditioning each prediction on the encoder's summary vector and on whatever it has generated so far.

This architecture was a genuine breakthrough β€” it let a single, fairly general neural network architecture handle sequence-to-sequence tasks without task-specific engineering, and it's the direct ancestor of the encoder-decoder Transformer variants covered later in this book. But it has an obvious structural weakness, which is exactly why the next chapter exists: the entire input sequence, no matter how long, has to be compressed into one fixed-size vector before the decoder ever starts generating. For a five-word sentence, this is a minor issue. For a fifty-word sentence, it becomes a serious information bottleneck β€” details from early in a long input sequence have to survive being compressed through the same fixed-size vector as everything else, and in practice, translation quality measurably degrades as input sentences get longer. This encoder-decoder bottleneck is the specific, concrete problem that attention mechanisms (Chapter 1.5) were invented to solve, and it's worth holding onto this exact failure mode, because attention's original motivation was not "let's build better language models" in the abstract β€” it was "let's stop forcing the entire input into one fixed-size vector."

6. Interview angle

RNNs come up constantly in interviews, both as history and as a way to test whether you understand why they were eventually abandoned for most large-scale language modeling:

  • "Why do vanilla RNNs struggle with long sequences?" β€” the expected answer is specifically about vanishing gradients during backpropagation through time, not a vague "they forget things."
  • "How does an LSTM solve the vanishing gradient problem, mechanically?" β€” a strong answer names the additive cell-state update path and the forget/input/output gates, and explains why additive updates preserve gradient magnitude better than repeated multiplicative updates.
  • "What is the encoder-decoder bottleneck, and how was it eventually solved?" β€” this is often used as a bridge question into attention and Transformers; the expected answer identifies the single fixed-size vector as the bottleneck and names attention as the fix.
  • "Why did the field move away from RNNs toward Transformers, given RNNs can in principle model arbitrarily long dependencies?" β€” tests whether you understand that the issue wasn't just modeling capacity but also parallelism: RNNs process tokens sequentially, one at a time, which is slow to train at scale β€” a theme picked up explicitly in Chapter 2.1.

7. Self-check questions

  1. Write the general recurrence relation for an RNN's hidden state update, and explain what property this gives you that fixed-window models (n-grams, Bengio's feedforward LM) cannot.
  2. Explain the vanishing gradient problem in your own words: what is being multiplied repeatedly during backpropagation through time, and why does that cause distant gradients to shrink?
  3. Describe the three gates in an LSTM and what each one controls. Which part of the LSTM's design specifically addresses vanishing gradients, and why?
  4. How does a GRU differ structurally from an LSTM, and what motivated that simplification?
  5. Describe the seq2seq encoder-decoder architecture and explain precisely where the "bottleneck" is β€” what specifically has to happen for a very long input sentence that doesn't have to happen for a short one?
  6. Why was solving the encoder-decoder bottleneck, rather than simply building deeper or larger RNNs, the more productive direction for the field to pursue? (This should prime your reading of Chapter 1.5.)

8. Sources

  • Elman, J. L. (1990). Finding Structure in Time. Cognitive Science, 14(2), 179–211. gwern.net PDF
  • Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780. MIT Press
  • Cho, K., van MerriΓ«nboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. arXiv:1406.1078
  • Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. arXiv:1409.3215
  • Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

In an RNN, what makes the hidden state $h_i$ different from the fixed context used by n-gram models or Bengio's feedforward LM?

Explanation: h_i = f(h_{i-1}, x_i) compresses an arbitrarily long history into a fixed-size summary vector, updated one token at a time β€” unlike a hard window cutoff.

What causes the vanishing gradient problem in vanilla RNNs during backpropagation through time?

Explanation: Because the same weight matrix is applied at every time step, the gradient involves a product of many copies of it β€” if the relevant values are consistently below 1, that product shrinks exponentially as you go back in time.

Mechanically, why does an LSTM's cell state help gradients survive over many time steps, compared to a vanilla RNN's hidden state?

Explanation: An additive update path doesn't get squashed by the repeated-multiplication effect that kills vanilla RNN gradients, letting the forget gate learn to keep relevant information open across long stretches.

What is the encoder-decoder bottleneck in a seq2seq RNN model?

Explanation: For long sentences, details from early in the input have to survive being squeezed through the same fixed-size vector as everything else β€” this is exactly the problem attention (Chapter 1.5) was invented to solve.

Suppose a vanilla RNN's backpropagated gradient shrinks by a factor of 0.9 at every time step. After 50 time steps, what fraction of the original gradient magnitude remains? (Round to 4 decimal places.)

Explanation: 0.9^50 β‰ˆ 0.00515 β€” the gradient has shrunk to roughly half a percent of its original size, illustrating the vanishing gradient problem concretely.

An LSTM has how many learned gates controlling its cell state in total (forget, input, and output combined)?

Explanation: Forget gate, input gate, and output gate β€” three gates in total.

A GRU simplifies the LSTM's three-gate design down to how many gates?

Explanation: A GRU uses an update gate and a reset gate β€” two gates β€” and drops the separate cell state entirely.

Suppose a different RNN's backpropagated gradient grows by a factor of 1.5 at every time step (exploding gradients). After 20 time steps, what is the gradient magnitude multiplied by, relative to its original value? (Round to the nearest whole number.)

Explanation: 1.5^20 β‰ˆ 3325 β€” the gradient has grown over three thousand times larger, illustrating why exploding gradients need to be controlled (e.g., via gradient clipping).

Why is truncated BPTT used instead of full backpropagation through time when training on long sequences?

Explanation: Backpropagating through an entire long sequence means holding every intermediate activation until the backward pass reaches it. Truncated BPTT runs the recurrence forward normally but only backpropagates through a fixed recent window, trading some long-range gradient signal for a step that fits in memory and finishes in reasonable time.

When is it appropriate to use a bidirectional RNN instead of a left-to-right one?

Explanation: A bidirectional RNN runs one RNN left-to-right and another right-to-left over the same sequence, concatenating their hidden states so every position sees both directions of context β€” which only makes sense when the whole input is already known, as with a seq2seq encoder, not with a genuine step-by-step generator.