Chapter 2.3 β€” Architectural Variants

Contents

  1. One set of parts, three shapes
  2. Encoder-only: BERT and bidirectional understanding
  3. Decoder-only: GPT and the causal generative shape
  4. Encoder-decoder: T5, BART, and sequence-to-sequence
  5. Why decoder-only won the scaling race
  6. Interview angle
  7. Self-check questions
  8. Sources

1. One set of parts, three shapes

Chapters 2.1 and 2.2 built a complete toolkit: scaled dot-product attention, multi-head projections, residual sublayers, and a way to inject positional information, whether additively at the input or directly inside attention. What Chapter 2.1 presented as "the Transformer" was actually a specific assembly of these parts β€” an encoder stack and a decoder stack, connected by cross-attention, built for machine translation. But nothing about the underlying toolkit forces that particular assembly. The same self-attention mechanism, the same feed-forward sublayers, the same residual-and-normalization pattern can be arranged in at least three structurally distinct ways, and the choice of arrangement turns out to matter enormously for what a model is good at and how it gets used.

This chapter is about those three shapes: encoder-only, decoder-only, and encoder-decoder. Understanding them isn't just architectural trivia β€” it's understanding a genuine fork in the road that the field took, and being able to explain, with real mechanistic reasons rather than folklore, why the decoder-only shape became the dominant one for the large-scale generative models that this book is centrally about. That explanation is where the chapter is heading, but it's worth walking through all three shapes on their own terms first, because each one made real, historically important gains before the field converged on the one that scales best.

2. Encoder-only: BERT and bidirectional understanding

The encoder half of the original Transformer, on its own, is a stack of self-attention and feed-forward sublayers where every position attends to every other position in the sequence with no masking at all β€” fully bidirectional, in the sense that a token's representation is built from context on both its left and its right simultaneously. Devlin et al.'s BERT, published in 2018, is built from exactly this encoder stack, with no decoder at all, and its central contribution was recognizing that this bidirectionality is extremely valuable for building rich representations of text, but is fundamentally incompatible with next-token prediction as a pretraining objective. If a model can see both left and right context for every position, then predicting a token from its full bidirectional context is trivial and useless as a training signal β€” the model would just be allowed to "look at the answer."

BERT's solution is the masked language modeling objective: randomly replace some fraction of input tokens (15% in the original paper) with a special mask token, and train the model to predict the original token at each masked position, using the full bidirectional context around it. This gives the encoder a genuine prediction problem to solve β€” the answer isn't directly visible β€” while still letting every unmasked position see context from both directions, which is exactly the representational strength that makes BERT-style models good at tasks like classification, entity extraction, or any task where you need a rich, context-aware representation of an existing piece of text rather than the ability to generate new text. Because there's no autoregressive generation involved at all, encoder-only models are typically used by taking their contextual output representations and feeding them into a small task-specific head, rather than by sampling text from them token by token β€” they're built to understand, not to write.

3. Decoder-only: GPT and the causal generative shape

The decoder-only shape strips the original Transformer down differently: keep the decoder's masked self-attention sublayers, drop the encoder and cross-attention sublayers entirely, since there's no separate source sequence to attend to, and use the resulting stack directly on a single sequence, predicting each token from everything before it. This is precisely the chain-rule decomposition from Chapter 1.1, $P(x_1, \ldots, x_n) = \prod_i P(x_i \mid x_1, \ldots, x_{i-1})$, implemented directly as a single causally masked Transformer stack with no auxiliary encoder at all. Radford et al.'s GPT-1 in 2018, and its much larger successors, are built exactly this way: masked self-attention layers stacked on top of each other, trained with the single, uniform objective of predicting the next token given everything before it.

The appeal of this shape is its directness. Because the causal masking from Chapter 2.1 already guarantees that position $i$ never sees position $j > i$ β€” $\text{mask}(i,j) = 0$ if $j \le i$, and $-\infty$ if $j > i$ β€” a decoder-only model can be trained on an entire document at once, with every position's next-token prediction computed in parallel and contributing to the loss simultaneously β€” there's no need for a separate encoding pass over a fixed source sequence, because there is no separate source sequence; the "source" and "target" are the same sequence, just shifted by one position. This uniformity turns out to matter far beyond just a training convenience, and is central to the story in the next section: because pretraining and generation both use exactly the same causal, autoregressive process, a decoder-only model trained purely on next-token prediction is already, mechanically, doing the same thing at inference time that it did during training, with none of the awkward mismatch that comes from needing a separate mechanism to bridge an encoding phase into a generation phase. GPT-3, described in Brown et al.'s 2020 paper, is the demonstration that scaling this exact shape β€” more layers, more parameters, more data, same causal next-token objective β€” produces a model capable of few-shot task performance from nothing but a prompt, without any task-specific fine-tuning at all, which is arguably the single result that set the trajectory for the rest of the LLM era.

4. Encoder-decoder: T5, BART, and sequence-to-sequence

The encoder-decoder shape keeps the full original structure from Chapter 2.1 intact: a bidirectional encoder builds a representation of an input sequence, and a causally masked decoder generates an output sequence, attending back into the encoder's representation at every layer via cross-attention. This shape is the most natural fit for tasks that are genuinely sequence-to-sequence in character β€” translation, obviously, since that's what motivated the original Transformer, but also summarization, where a long document needs to be fully absorbed bidirectionally before a shorter output is generated from it, or any task with a clean separation between "the thing you're conditioning on" and "the thing you're producing."

Raffel et al.'s T5 pushed this shape furthest by proposing that essentially every NLP task β€” classification, translation, summarization, question answering β€” could be cast into the same text-to-text format, with the specific task described as part of the input text itself, and handled by one single encoder-decoder architecture and training recipe rather than a different architecture per task. Lewis et al.'s BART took a related but distinct pretraining angle: corrupt an input document with a variety of noising functions (token masking, sentence permutation, text infilling, and others), and train the encoder-decoder to reconstruct the original, uncorrupted document, which combines BERT-like bidirectional understanding of the corrupted input with GPT-like autoregressive generation of the reconstruction, in one architecture. Both of these models demonstrate a genuine strength of the encoder-decoder shape: the input doesn't need to be autoregressively constrained at all, so the encoder can build the richest possible bidirectional representation of it, while the decoder still gets the autoregressive structure it needs for coherent, well-formed generation, with cross-attention as the bridge between the two.

5. Why decoder-only won the scaling race

Given that the encoder-decoder shape has this seemingly strict architectural advantage β€” bidirectional understanding of the input, plus autoregressive generation of the output β€” the decoder-only shape's dominance in modern large-scale LLMs might look surprising at first. The reasons come down to a handful of practical, scaling-relevant considerations that matter more and more as model and data size grow, rather than any claim that decoder-only models are representationally superior task for task.

The first is uniformity of the pretraining objective, which was already touched on in Section 3 but deserves stating as a scaling argument specifically: a decoder-only model has exactly one thing to learn to do, next-token prediction over an undifferentiated stream of text, and that one objective is what it's trained on and what it's used for at inference time. An encoder-decoder model has to be trained with some choice of denoising or corruption scheme for the encoder side, plus a generation objective for the decoder side, and those choices introduce design decisions β€” how to corrupt, how much, with what noise function β€” that don't obviously get more effective just because you throw more data and parameters at them, and that create a train/inference mismatch of their own (the model never sees clean input at inference the way encoder-only pretraining of the corrupted side might imply). A single, simple objective, applied identically to every token in every document, is a much easier thing to scale up predictably, and predictable scaling behavior is exactly what the field increasingly organizes around (a theme this book returns to directly in the chapters on scaling laws).

The second is parameter and compute efficiency for a fixed capability target. An encoder-decoder model pays for two separate stacks (plus cross-attention), which for a fixed total parameter budget means each stack individually is roughly half the size it could have been if all parameters were pooled into one stack. For generative tasks specifically β€” the task class that turned out to matter most once in-context learning and instruction-following emerged as capabilities β€” the encoder's separate bidirectional processing of a fixed input isn't obviously buying enough to justify permanently reserving half the model's capacity for it, especially once prompts themselves become the mechanism for providing whatever "input context" a task needs, rather than a structurally separate encoder pass.

The third, and arguably the most consequential in retrospect, is that decoder-only models handle variable and effectively unbounded numbers of tasks through a single unified interface β€” the prompt β€” without any architectural change at all, simply by conditioning the same next-token prediction machinery on whatever text precedes the generation point. An encoder-decoder model's hard separation between "input" and "output" sequences maps naturally onto a fixed, well-defined sequence-to-sequence task, but maps awkwardly onto the open-ended, in-context, multi-turn, tool-using, instruction-following usage patterns that came to define how the largest LLMs are actually deployed, where what counts as "input" and what counts as "generated so far" blur together and change turn by turn. A decoder-only model doesn't need to decide in advance where that boundary is, because its entire sequence, prompt and generation alike, is processed by the same causal mechanism throughout β€” which is a genuine architectural fit to the way the field's usage of these models evolved, not merely a historical accident.

It's worth naming a fourth, intermediate shape here too, since it complicates the clean three-way split section 1 opened with: prefix-LM. A prefix-LM model is architecturally a single decoder-only stack β€” no separate encoder, no cross-attention β€” but its attention mask is modified so that a designated prefix portion of the sequence (the input or context) is attended to bidirectionally by every token, including other prefix tokens, while the remainder of the sequence (the part being generated) keeps the ordinary causal mask that only looks backward. This gets you the bidirectional-context benefit of an encoder for the fixed input portion without paying for a structurally separate stack at all. Tay et al.'s UL2 pushes this further with a mixture-of-denoisers pretraining objective that blends span-corruption-style denoising (the kind of objective that motivates an encoder-decoder shape), prefix-LM-style objectives, and ordinary causal-LM objectives, all trained on the same single decoder-only stack β€” the practical lesson being that pretraining objective and architectural shape are more separable dimensions than sections 2 through 4 might suggest in isolation, and prefix-LM is the mechanism that makes that separation possible.

None of this means encoder-only or encoder-decoder shapes became useless β€” encoder-only models remain a strong, efficient choice for pure representation tasks where no generation is needed at all, and encoder-decoder models retain real advantages for genuinely fixed-shape sequence-to-sequence problems. But for the specific trajectory this book is tracing β€” training the largest possible general-purpose generative models β€” the decoder-only shape's combination of objective uniformity, full-capacity allocation to one stack, and a single interface that scales to arbitrary tasks without architectural change is what won out. That's the shape assumed for essentially every chapter from here forward, and it's exactly what raises the next question this book has to answer: a decoder-only stack of the kind described in this chapter is not yet stable to train at real depth without help, and the specific mechanism that makes very deep stacks trainable at all β€” normalization β€” is where the next chapter picks up.

6. Interview angle

"Why can't you just train an encoder-only model like BERT to generate text autoregressively?" A strong answer identifies the structural incompatibility directly: encoder self-attention is unmasked and bidirectional by construction, so every position can already see the entire sequence, including positions that would need to be "future" tokens during generation. There's no causal masking to prevent a token from attending to itself or later tokens, so there's no way to run it left-to-right as a generator without changing the attention pattern itself, at which point it's no longer the same architecture β€” it becomes a masked decoder.

"Why does BERT need the masked language modeling objective specifically, rather than plain next-token prediction?" The answer should explain that bidirectional self-attention makes next-token prediction trivial and uninformative, because the token to be predicted is already visible in the unmasked context. Masking a subset of input tokens and asking the model to reconstruct them from bidirectional context around the gaps creates a genuine prediction problem while preserving the bidirectionality that's the whole point of using an encoder-only architecture.

"Give a mechanistic, not folklore, explanation for why decoder-only became the dominant shape for large LLMs." A strong answer resists the vague "it's simpler" answer and instead lists concrete reasons: a single, uniform pretraining objective (next-token prediction) that matches the inference-time computation exactly, with no train/inference mismatch; full parameter budget allocated to one stack rather than split across encoder and decoder; and, most importantly, a single sequential interface (the prompt) that accommodates arbitrarily varied tasks without any architectural change, which matches how usage of large LLMs actually evolved toward open-ended, multi-turn, in-context interaction.

"When would an encoder-decoder architecture still be the right choice today?" A strong answer notes that tasks with a clean, fixed separation between a bounded input and a structurally different output β€” machine translation, certain summarization or structured extraction pipelines β€” can still benefit from a dedicated bidirectional encoder pass over the input, paired with a smaller, focused decoder, and that this can be more parameter-efficient than routing the same task through a much larger decoder-only model whose full capacity has to be general-purpose. The strong answer avoids implying encoder-decoder models are simply obsolete.

"What specifically breaks if you remove the causal mask from a decoder-only model during pretraining, on the same data and objective?" A strong answer explains that removing the mask lets each position see tokens after it, which means the loss for predicting position $i$ can be trivially minimized by copying from position $i$'s own future context rather than learning to predict it from the past β€” the model would learn to "cheat" during training and would be uncalibrated (effectively untrained for the task) at inference, where future tokens genuinely don't exist yet.

7. Self-check questions

  1. What are the three structural shapes a Transformer's attention and feed-forward sublayers can be assembled into, and what's the one component that differs mechanically between encoder-only and decoder-only self-attention?
  2. Why is next-token prediction an unusable pretraining objective for a purely bidirectional encoder stack, and what objective does BERT use instead to work around this?
  3. Explain, mechanistically, why a decoder-only model's pretraining computation and its inference-time computation are essentially the same process, in a way that isn't true for an encoder-decoder model.
  4. What does T5's text-to-text framing achieve, conceptually, relative to having a separate architecture per task?
  5. List the three scaling-relevant reasons given in this chapter for why decoder-only architectures became dominant, and explain which one you find most compelling and why.
  6. Under what circumstances would you still recommend an encoder-decoder architecture over a decoder-only one for a real production task?
  7. How does cross-attention, as introduced in Chapter 2.1, serve as the structural bridge between the encoder and decoder stacks in models like T5 and BART?

8. Sources

  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
  • Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training (GPT-1). OpenAI technical report. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
  • Brown, T. B. et al. (2020). Language Models are Few-Shot Learners (GPT-3). NeurIPS 2020. arXiv:2005.14165. https://arxiv.org/abs/2005.14165
  • Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5). JMLR 2020. arXiv:1910.10683. https://arxiv.org/abs/1910.10683
  • Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019). BART. ACL 2020. arXiv:1910.13461. https://arxiv.org/abs/1910.13461
  • Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Bahri, D., Schuster, T., Zheng, S., Zhou, D., Houlsby, N., & Metzler, D. (2022). UL2: Unifying Language Learning Paradigms. arXiv:2205.05131. https://arxiv.org/abs/2205.05131

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

Why can't a pure encoder-only model (like BERT) be trained directly on next-token prediction, the way a decoder-only model is?

Explanation: There's no causal mask preventing a position from seeing itself or later tokens, so next-token prediction would be trivial and uninformative β€” the model could just "look at the answer."

What problem does BERT's masked language modeling objective solve?

Explanation: Masking a subset of tokens (15% in the original paper) and asking the model to reconstruct them from bidirectional context creates a real prediction problem while preserving bidirectionality.

According to this chapter, what is the single most consequential reason decoder-only models became dominant for large-scale generative LLMs?

Explanation: An encoder-decoder model's hard input/output separation maps awkwardly onto open-ended, multi-turn, in-context usage β€” a decoder-only model doesn't need to decide in advance where that boundary is.

What is T5's core contribution, conceptually?

Explanation: T5 reframes classification, translation, summarization, and QA all as text-to-text problems, with the task itself described in the input text, using one architecture rather than one per task.

Using BERT's original masking rate of 15%, how many tokens would be masked in a 200-token input sequence?

Explanation: 200 Γ— 0.15 = 30 tokens.

A fixed total parameter budget of 12 billion is split evenly between an encoder-decoder model's two stacks (ignoring cross-attention parameters). How many billion parameters does each stack get?

Explanation: 12 / 2 = 6 billion parameters per stack β€” this is the "each stack is roughly half the size it could have been" tradeoff the chapter describes.

A decoder-only model is trained on a document of 2048 tokens. Because every position's next-token prediction is computed in parallel and contributes to the loss, how many total next-token predictions (one per position, per the chain rule from Chapter 1.1) does this single document contribute?

Explanation: The chain rule decomposes P(x_1,...,x_n) into n conditional predictions, one per position β€” here, n = 2048.

Suppose an encoder-decoder Transformer's encoder stack has 12 layers and its decoder stack also has 12 layers. How many total layers does the full model have (encoder + decoder combined)?

Explanation: 12 + 12 = 24 layers total, split evenly between the two stacks.