Chapter 1.5 β€” Attention Before Transformers

Contents

  1. The problem, restated precisely
  2. Bahdanau attention: let the decoder look back
  3. How alignment scores are computed and used
  4. Luong attention: a simpler, more general recipe
  5. What attention changed, and what it didn't
  6. Interview angle
  7. Self-check questions
  8. Sources

1. The problem, restated precisely

Chapter 1.4 ended with a specific, concrete failure mode: in a sequence-to-sequence model, the encoder has to compress an entire input sentence into a single fixed-size vector before the decoder generates anything, and that compression becomes an information bottleneck as sentences get longer. It's worth being precise about why this is a bottleneck and not just a vague inefficiency: at every single step of decoding, the decoder is working from the exact same summary vector, regardless of which part of the input sentence is actually relevant to the word it's generating right now. Translating the fifth word of a long sentence and the fiftieth word both draw on identical, undifferentiated information about the whole input.

The fix, introduced independently in very similar forms by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, and shortly after by Minh-Thang Luong, Hieu Pham, and Christopher Manning, is to stop compressing the input into one vector at all. Instead, keep the encoder's hidden state at every input position, and let the decoder, at each step of generation, compute a weighted combination of all of those hidden states β€” a combination that changes at every decoding step, so the decoder can "look at" different parts of the input depending on what it's currently generating. This is attention, and it is the single most consequential idea in this book, because the mechanism introduced here to fix one specific translation problem is, essentially unchanged in its core mathematics, the operation that Transformers are built entirely out of.

2. Bahdanau attention: let the decoder look back

Bahdanau et al.'s 2014 paper (published at ICLR 2015) frames the fix as "jointly learning to align and translate" β€” the phrase "align" is doing real work in that title. Instead of a single encoder summary vector, the encoder (a bidirectional RNN, so each position's hidden state reflects context from both directions) produces one hidden state per input token: \(h_1, h_2, \ldots, h_T\).

At each decoding step \(i\), the decoder doesn't just consume a single fixed context vector β€” it computes a fresh one, specific to that step, as a weighted sum over all the encoder hidden states:

$$c_i = \sum_{j=1}^{T} \alpha_{ij} h_j$$

The weights \(\alpha_{ij}\) (which sum to 1 across \(j\), via a softmax) are the "attention weights" β€” how much decoding step \(i\) should attend to input position \(j\). Critically, these weights are computed dynamically from the current decoder state and each encoder hidden state, meaning the model learns, as part of training, which input positions matter for which output positions β€” this is the "alignment" the title refers to, and for translation specifically, these learned alignments often correspond strikingly well to actual word-for-word or phrase-for-phrase correspondences between the source and target languages, without ever being told explicitly what those correspondences are.

3. How alignment scores are computed and used

The weights \(\alpha_{ij}\) come from normalizing a set of raw alignment scores \(e_{ij}\), one per encoder position, computed by comparing the decoder's current state to each encoder hidden state:

$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T} \exp(e_{ik})}, \qquad e_{ij} = a(s_{i-1}, h_j)$$

where \(s_{i-1}\) is the decoder's hidden state from the previous step, and \(a\) is a small learned function β€” in Bahdanau's original formulation, a single-hidden-layer feedforward network β€” that scores how well encoder position \(j\) matches what the decoder currently needs. This is worth breaking down carefully because this exact three-part structure β€” compute a compatibility score between a "query" (the decoder's current state) and a set of "keys" (the encoder states), normalize those scores with a softmax, and use the result to weight a set of "values" (again the encoder states) β€” is precisely the structure of the attention mechanism inside every Transformer discussed later in this book. The query/key/value terminology hadn't been introduced yet in 2014, and the scoring function was more elaborate than what came later, but the underlying operation β€” a differentiable, learned, softmax-weighted lookup over a set of vectors β€” is identical.

The context vector \(c_i\) produced this way is then combined with the decoder's own state to produce the actual output prediction for step \(i\). The whole thing β€” encoder, alignment scoring, weighted combination, decoder β€” is trained end-to-end with ordinary backpropagation, exactly like the rest of the models in this book; nothing about the attention weights is hand-specified or supervised separately.

4. Luong attention: a simpler, more general recipe

Luong, Pham, and Manning's 2015 paper arrived at a very similar idea via a somewhat different, more systematic framing, and its main contribution was to simplify and generalize the scoring function \(a(s, h)\), while also proposing "global" versus "local" attention variants (whether to attend over the whole input sequence or a restricted window around a predicted position).

Rather than Bahdanau's small feedforward network, Luong et al. proposed and compared several simpler scoring functions, most notably a dot product between the decoder and encoder states (\(s^\top h\)) and a "general" bilinear form (\(s^\top W h\), with a learned weight matrix \(W\)). The dot-product variant in particular is significant far beyond this chapter: it is, up to a scaling factor, exactly the compatibility function used inside scaled dot-product attention in the Transformer β€” \(\frac{QK^\top}{\sqrt{d_k}}\), with the scaling factor \(\sqrt{d_k}\) added for the numerical-stability reasons Chapter 2.1 covers in detail. Luong's paper also clarified an architectural detail that subsequent work adopted widely β€” computing the attention context vector and combining it with the decoder's hidden state before making the output prediction for that step, rather than folding attention into the recurrence itself, which made the mechanism easier to reason about and to plug into different decoder designs.

The practical result of both papers was the same: substantially better machine translation quality, especially on longer sentences, precisely because the encoder-decoder bottleneck was gone β€” the decoder could always consult the specific input positions relevant to whatever it was generating, rather than working from one compressed summary for the entire sentence.

5. What attention changed, and what it didn't

It's worth being precise about the scope of what attention fixed at this stage in the book's history, because the next chapter's leap is bigger than "attention, but better," and understanding the boundary here makes that leap clearer.

What attention fixed: the encoder-decoder bottleneck specifically. The decoder no longer has to route everything it needs through a single fixed-size vector; it can dynamically pull information from wherever in the input sequence is relevant to the current output step.

What attention didn't fix, in these 2014–2015 formulations: the underlying architecture was still built on RNNs. The encoder still processed its input one token at a time, sequentially, meaning training still couldn't be parallelized across positions within a sequence, and the vanishing-gradient-related difficulties of very long sequences (Chapter 1.4) were softened but not eliminated, since the recurrence itself was untouched β€” attention added a way to reach back to any encoder position, but computing those encoder positions in the first place still required marching through the sequence step by step.

Bahdanau's and Luong's formulations are the two that matter most for tracing the direct line to Transformers, but they were the opening move in a much larger family of attention variants β€” different scoring functions, attention over memory banks rather than just an encoder's states, hierarchical attention over words and then sentences β€” that flourished in NLP research through the mid-2010s; Galassi et al.'s survey is a good map of that broader landscape if you want it, though this book's throughline only needs the two formulations covered above.

This sets up the exact question the next chapter answers: what if you didn't need the recurrence at all? What if attention β€” the mechanism that computes a weighted combination over a set of vectors based on learned compatibility scores β€” could be used not just to let a decoder look back at an encoder, but as the entire mechanism by which a model builds contextual representations of a sequence, discarding recurrence altogether? That question, and its answer, is "Attention Is All You Need" β€” the title of the paper that opens Part II of this book, and not a coincidental echo of what you've just read.

6. Interview angle

This chapter is a favorite for testing whether a candidate understands attention as an idea, rather than as "the thing inside Transformers":

  • "What specific problem does attention solve, and what mechanism did it replace?" β€” the expected answer names the encoder-decoder bottleneck directly, not a vague "attention lets the model focus on important things."
  • "Walk me through how an attention weight is actually computed." β€” this is checking whether you can state the query/compatibility-score/softmax/weighted-sum pipeline concretely, ideally using the query-key-value framing even for the pre-Transformer version.
  • "What's the difference between Bahdanau and Luong attention?" β€” a strong answer identifies the scoring function (feedforward network vs. dot product/bilinear) as the main distinction, and can explain why the dot-product version foreshadows scaled dot-product attention.
  • "Why wasn't RNN-plus-attention enough β€” why did the field move to a fully attention-based architecture?" β€” the expected answer is about parallelism and training efficiency at scale, not just modeling quality; this is the direct bridge into Chapter 2.1.

7. Self-check questions

  1. Precisely describe the encoder-decoder bottleneck from Chapter 1.4, and explain exactly what attention changes about the decoder's access to encoder information.
  2. Write out the equation for a context vector \(c_i\) as a weighted sum of encoder hidden states, and explain where the weights \(\alpha_{ij}\) come from.
  3. What are the three components of the general attention pattern (as it would later be called: query, key, value), and how do they map onto Bahdanau's decoder state, encoder states, and encoder states again?
  4. Compare Bahdanau's feedforward scoring function to Luong's dot-product scoring function. Why does the dot-product version matter more for the rest of this book?
  5. What did attention (in its 2014–2015 RNN-based form) fail to fix about the underlying architecture? Be specific about what still required sequential computation.
  6. Why is "discard the recurrence and keep only the attention mechanism" a coherent next step, rather than an unrelated new idea? Try to state, in one or two sentences, what you expect a fully attention-based architecture would need to add to replace what recurrence used to provide.

8. Sources

  • Bahdanau, D., Cho, K., & Bengio, Y. (2014/2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv:1409.0473
  • Luong, M.-T., Pham, H., & Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP 2015. arXiv:1508.04025
  • Galassi, A., Lippi, M., & Torroni, P. (2020). Attention in Natural Language Processing. IEEE Transactions on Neural Networks and Learning Systems. arXiv:1902.02181

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

What specific problem does attention (Bahdanau/Luong) solve in a seq2seq RNN model?

Explanation: The decoder no longer has to route everything through one fixed-size vector β€” it can dynamically pull information from whichever input positions are relevant to the current output step.

How does Luong's dot-product scoring function relate to what comes later in the book?

Explanation: Luong's dot-product score sΒ·h foreshadows the QKα΅€ term in scaled dot-product attention β€” the same core operation, just without the √d_k scaling and the learned Q/K/V projections.

What did attention (in its 2014–2015 RNN-based form) fail to fix about the underlying architecture?

Explanation: Attention added a way to reach back to any encoder position, but computing those encoder positions in the first place still required marching through the sequence step by step β€” that's exactly what Chapter 2.1 removes.

In the query/key/value framing applied retroactively to Bahdanau attention, what plays the role of the "query"?

Explanation: The decoder's state is what's "looking for" relevant information β€” the query β€” while the encoder hidden states serve as both the keys (for scoring) and the values (for the weighted sum).

Given encoder hidden states (treated as scalars) $h_1 = 2$, $h_2 = 4$, $h_3 = 6$ and attention weights $\alpha_{i1} = 0.1$, $\alpha_{i2} = 0.3$, $\alpha_{i3} = 0.6$, compute the context vector $c_i = \sum_j \alpha_{ij} h_j$.

Explanation: c_i = 0.1(2) + 0.3(4) + 0.6(6) = 0.2 + 1.2 + 3.6 = 5.0.

Raw alignment scores for three encoder positions are $e_{i1} = 1$, $e_{i2} = 2$, $e_{i3} = 3$. After applying the softmax from the attention-weight formula, what is $\alpha_{i3}$ (the weight for the third position)? (Round to 2 decimal places.)

Explanation: softmax(1,2,3) = (e^1, e^2, e^3) / (e^1+e^2+e^3) β‰ˆ (2.72, 7.39, 20.09) / 30.19, so Ξ±_{i3} β‰ˆ 20.09/30.19 β‰ˆ 0.67.

Using Luong's dot-product scoring function $s^\top h$, with decoder state $s = (1, 2, 3)$ and encoder hidden state $h = (4, 0, 1)$, what is the raw alignment score?

Explanation: sΒ·h = 1(4) + 2(0) + 3(1) = 4 + 0 + 3 = 7.

Using Luong's "general" bilinear scoring function $s^\top W h$, with $s = (1, 1)$, $h = (2, 1)$, and $W = \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}$, compute the score.

Explanation: Wh = (2(2)+0(1), 0(2)+3(1)) = (4, 3). Then sΒ·(Wh) = 1(4) + 1(3) = 7.