Chapter 4.2 β€” Pretraining Objectives and Data

Contents

  1. From tokens to training signal
  2. Three families of pretraining objective, revisited
  3. Why decoder-only causal language modeling won
  4. Data quality and composition matter as much as architecture
  5. Deduplication as a concrete, measured intervention
  6. Curating web-scale data: RefinedWeb and FineWeb
  7. Interview angle
  8. Self-check questions
  9. Sources

1. From tokens to training signal

Chapter 4.1 established how raw text becomes a sequence of discrete tokens, and closed on the observation that even a well-designed tokenizer only gets you as far as a sequence of integers β€” it says nothing about what you actually train a model to predict, or what text you train it on. Those two questions turn out to be tightly linked, because different pretraining objectives make different architectural commitments about what "context" a token is allowed to see, and Chapter 2.3's survey of encoder-only, decoder-only, and encoder-decoder architectures already laid the groundwork for exactly this distinction. This chapter revisits that architectural material through the lens of training objective, and then turns to a question that gets comparatively little attention in most treatments of transformers but drives an enormous amount of real engineering effort in practice: what text, and how much cleaning of that text, actually produces a good model.

2. Three families of pretraining objective, revisited

The causal (autoregressive) language modeling objective, the one this book has used since Chapter 1.1's chain-rule decomposition, trains a model to predict each token using only the tokens that came before it, left to right, exactly matching the decoder-only architecture's use of a causal attention mask. Masked language modeling, popularized by BERT, instead corrupts a fraction of the input tokens β€” typically by replacing them with a special mask token, though some are left unchanged or replaced with a random token as a regularization trick β€” and trains the model to predict the original identity of each corrupted position using full bidirectional context, both the tokens before and after it. This requires an encoder-only architecture with no causal masking, since the whole point is to let every position see the entire sequence. Span corruption, used by T5-style encoder-decoder models, generalizes masked language modeling by corrupting contiguous spans of tokens rather than isolated positions and training the decoder to reconstruct the missing spans autoregressively, conditioned on the encoder's bidirectional representation of the corrupted input β€” an objective often described as a "prefix LM" or denoising objective sitting between the other two.

These are not three arbitrary options; each is the natural training objective for the corresponding architecture's information-flow constraints, and each produces a model well-suited to different downstream uses. Masked language models produce excellent representations for classification and understanding tasks where you have the whole input available at once and just need a good encoding of it. Encoder-decoder models with span corruption are a natural fit for genuinely sequence-to-sequence tasks like translation or summarization, where there's a clear source and target. Causal language models are suited to open-ended generation, since generation is inherently a left-to-right process: you don't have the rest of the sequence available when you're producing the first token of a reply.

3. Why decoder-only causal language modeling won

Given that all three objectives have reasonable use cases, it's worth being precise about why decoder-only causal pretraining became the dominant paradigm for general-purpose large language models specifically, rather than treating this as an accident of fashion. The central reason is consistency between training and inference. A causal language model is trained to do exactly the thing it will be asked to do at deployment time: given a prefix, predict what comes next, one token at a time, using only leftward context. There is no mismatch between the objective optimized during pretraining and the task performed at inference β€” generation via autoregressive sampling is just repeated application of the training objective. A masked language model, by contrast, is trained to fill in blanks given bidirectional context that simply does not exist at the moment of open-ended generation, since you don't have "the rest of the sentence" when you're writing the first word of a reply to a user; using such a model for open-ended generation therefore requires bridging an objective mismatch that causal models never incur in the first place.

This consistency matters practically, not just aesthetically. It means a single causal decoder can be prompted to do essentially any task that can be phrased as "continue this text" β€” question answering, summarization, translation, code generation, dialogue β€” without needing task-specific architectural changes, because the pretraining objective already is the general-purpose task. Encoder-only and encoder-decoder models can be adapted to generation, and were for years, but doing so well typically requires either giving up bidirectionality for the generated portion or engineering around the asymmetry between the two halves of the model. Once it became clear that a sufficiently large decoder-only model, trained purely on next-token prediction over enough diverse text, could match or exceed the downstream task performance of architectures explicitly designed around specific task formats, the field converged toward decoder-only causal pretraining as the default recipe for general-purpose models, reserving encoder-only and encoder-decoder architectures for settings β€” dense retrieval embeddings, certain translation pipelines β€” where their particular inductive biases still pay off.

4. Data quality and composition matter as much as architecture

It's tempting, having spent several chapters on architecture and objectives, to treat the training corpus as a fixed, given input β€” "the internet," roughly β€” and focus all engineering attention on the model. In practice, what corpus you train on, and how carefully you clean and curate it, has at least as large an effect on the resulting model's quality as any architectural choice at a comparable scale, and arguably a larger one, because bad data doesn't just fail to help β€” it actively teaches the model to reproduce its own defects. A corpus full of boilerplate, duplicated text, machine-generated spam, or low-information content wastes training compute on tokens that carry little useful signal, and a corpus with duplicated passages creates a specific, well-documented pathology: memorization.

5. Deduplication as a concrete, measured intervention

Lee et al. gave this concern empirical teeth. They showed that large web-scraped training corpora contain substantial amounts of near-exact and exact duplicate text β€” the same passage, or close variants of it, appearing many times across different documents, whether from mirrored web pages, quoted articles, or boilerplate license text β€” and that this duplication has two measurable costs. First, models trained on duplicated data are considerably more likely to memorize and regurgitate verbatim passages from their training set, which is both a quality problem (regurgitation looks like a failure of generalization, not a display of it) and, increasingly, a genuine privacy and copyright concern once you consider that memorized passages can include personal information or licensed text scraped incidentally. Second, and more directly relevant to a training-efficiency argument, duplicate text wastes compute: every time the model trains on a repeated passage, it spends a training step on text it has effectively already seen, rather than on tokens that would teach it something new. Lee et al. showed that deduplicating a training corpus β€” at the level of exact document matches and also near-duplicate matches found via techniques like suffix-array-based substring matching β€” measurably improves downstream model quality for a fixed amount of training compute, which is exactly the kind of result that makes deduplication a standard, non-optional step in modern large-scale pretraining pipelines rather than a nice-to-have.

6. Curating web-scale data: RefinedWeb and FineWeb

Deduplication is one specific, well-isolated intervention, but production data pipelines combine many such interventions: filtering out low-quality or unsafe content via heuristic rules (removing pages that are mostly boilerplate, non-prose, or flagged as adult or otherwise unsuitable content), training lightweight quality classifiers to score and filter documents by predicted usefulness, and deduplicating at multiple granularities from exact document matches down to overlapping substrings. For a long time, the strongest publicly known models leaned heavily on datasets blending web scrapes with curated, often proprietary sources β€” books, code repositories, carefully selected reference text β€” under the assumption that raw web-scraped text, however abundant, was simply too noisy to be the primary substrate for a frontier model on its own.

The RefinedWeb and FineWeb lines of work pushed back on that assumption directly. RefinedWeb, built to train the Falcon models, demonstrated that a large-scale pipeline applying careful filtering and deduplication to Common Crawl web data alone β€” without blending in curated proprietary corpora β€” could produce training data that matched or exceeded the downstream quality of datasets that did rely on such curated sources, essentially showing that the "curated corpus is necessary" assumption was an artifact of insufficiently rigorous web-data filtering rather than a fundamental limitation of web data itself. FineWeb extended this line of work with an even larger, more thoroughly documented and ablated web-scale dataset, systematically testing which filtering choices actually improved downstream benchmark performance rather than relying on filtering heuristics chosen by intuition alone, and released the resulting dataset and the accompanying analysis publicly. Together these two efforts reframed data curation from "assemble whatever curated sources you can license" to "apply rigorous, measurable filtering to the abundant raw web," which is now the dominant approach for large open pretraining corpora and a strong illustration of a broader theme this chapter has been building toward: at the scale modern LLMs are trained, data engineering is not an afterthought bolted onto model engineering, it is co-equal with it, and it is exactly as amenable to careful measurement and iteration as any architectural choice.

Filtering and deduplication decide what stays in the corpus; a separate question is how much of each remaining source to show the model, and that question turns out to be just as consequential. A pretraining corpus is never a single homogeneous pile of text β€” it's a mixture of domains (web pages, books, code, academic papers, dialogue) β€” and the obvious default, sampling each domain in proportion to how much of it you happen to have, is rarely the mixture that produces the best model, since a domain's raw abundance on the internet has little to do with how useful it is as training signal. Xie et al.'s DoReMi automates the search for a better mixture: train a small proxy model, use it to identify which domains a fixed reference model under-fits relative to their sampling weight, and use that signal to re-derive domain weights that a much larger model is then trained on β€” improving downstream performance over a hand-tuned mixture without ever having to train the large model more than once to find it. The general lesson generalizes past DoReMi's specific algorithm: domain weight is a real hyperparameter of pretraining, with a measurable effect on quality, and treating "the mixture" as fixed by whatever data happens to be lying around leaves real performance on the table.

Code is the sharpest illustration of why domain weight matters beyond simply "does this domain help the tasks it obviously resembles." A model trained with a meaningful fraction of source code in its pretraining mix reliably does better not only at code generation but at multi-step, structured reasoning tasks that have nothing to do with programming β€” arithmetic word problems, logical deduction, multi-hop question answering. Aryabumi et al.'s systematic study of this effect finds the improvement holds even after controlling for total data quality, and attributes it to code's unusually explicit structure: a training signal built from function calls, control flow, and variable state that has to compose correctly to run at all, which appears to transfer to other domains that reward step-by-step, compositional problem-solving. This is a concrete instance of the section's larger point β€” a domain's value as training data is not well predicted by how "relevant" it looks to your target tasks on the surface β€” and it foreshadows the more deliberate use of structured, verifiable problems that Chapter 5.1's reasoning-training discussion picks up.

Taken together, this chapter has argued that what a model is trained to predict and what it is trained on are inseparable engineering decisions, each with measurable consequences for the resulting model's downstream behavior. But so far every claim in this chapter has been qualitative β€” "more careful filtering helps," "deduplication helps" β€” without a way to predict how much any of these choices will help before you spend the compute to find out. That predictive question, whether you can forecast a model's loss and downstream performance as a function of its size, its data, and the compute you're willing to spend, before training it, is exactly what scaling laws were built to answer, and it's where the next chapter turns.

7. Interview angle

Q: Why did decoder-only causal language modeling become the standard pretraining approach for general-purpose LLMs, rather than masked language modeling or span corruption? A strong answer centers the argument on train/inference consistency: causal pretraining trains the model to do exactly the task it performs at deployment (predict the next token given a leftward prefix), so there is no objective mismatch to bridge for open-ended generation, whereas masked and span-corruption objectives rely on bidirectional context or an encoder/decoder split that isn't available when generating text one token at a time from a prompt. A strong answer also acknowledges that masked/encoder-decoder objectives remain well suited to their own use cases (embeddings, certain seq2seq tasks) rather than claiming decoder-only is universally superior.

Q: What specifically does deduplication of pretraining data protect against, and why does it matter for training efficiency, not just data hygiene? A strong answer names both consequences distinctly: increased verbatim memorization (a quality and privacy/copyright risk) and wasted compute, since training steps spent on duplicated text don't teach the model anything new relative to the first time it saw that text, so removing duplicates improves downstream quality per unit of compute rather than just being a cleanliness measure.

Q: RefinedWeb and FineWeb both argue web-scraped data can match curated corpora. What's the actual mechanism that makes this possible? A strong answer explains that it's rigorous, carefully ablated filtering and multi-granularity deduplication applied at scale to raw web crawls (like Common Crawl), not some inherent property of web text β€” the claim is that prior assumptions about needing curated proprietary sources reflected insufficiently careful web-data pipelines, and that with enough filtering rigor, the sheer scale and diversity of web data can produce comparable or better training data than smaller curated corpora.

Q: If you were told to improve a pretraining corpus with a fixed compute budget for training, what levers would you consider before touching model architecture? A strong answer lists deduplication at multiple granularities, quality filtering via heuristics or learned classifiers, and removal of unsafe/low-value content, and frames all of these as compute-efficiency interventions β€” the goal is maximizing useful signal per training token, not simply accumulating more raw text.

Q: Is masked language modeling "obsolete" now that decoder-only models dominate? A strong answer resists the trap of the question: masked language modeling remains the right objective for producing strong bidirectional representations for retrieval and classification embeddings, where the entire input is available at inference time and there's no generation step to be consistent with; "obsolete for generation" is different from "obsolete."

8. Self-check questions

  1. Why does a masked language model's training objective create an inherent mismatch with open-ended text generation, in a way that causal language modeling does not?
  2. Which architectural property from Chapter 2.3 does each of the three pretraining objectives (causal LM, masked LM, span corruption) require, and why?
  3. What are the two distinct harms Lee et al. attribute to duplicated text in pretraining corpora?
  4. Why is "the model memorized a training example verbatim" considered a failure rather than a success, given that language models are explicitly trained to predict text accurately?
  5. What did RefinedWeb and FineWeb argue against, and what evidence supported their position?
  6. If a corpus is deduplicated but otherwise unfiltered for quality, would you expect better or worse downstream performance than a filtered-but-not-deduplicated corpus of the same token count? What does your answer depend on?
  7. Why does this chapter frame data curation as "co-equal" with model architecture rather than a secondary concern?

9. Sources

  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
  • Raffel, C., Shazeer, N., Roberts, A., et al. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5). arXiv:1910.10683. https://arxiv.org/abs/1910.10683
  • Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). Not on arXiv. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
  • Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022). Deduplicating Training Data Makes Language Models Better. ACL 2022. arXiv:2107.06499. https://arxiv.org/abs/2107.06499
  • Penedo, G., Malartic, Q., Hesslow, D., et al. (2023). The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. NeurIPS 2023. arXiv:2306.01116. https://arxiv.org/abs/2306.01116
  • Penedo, G., Kydlíček, H., Lozhkov, A., et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS 2024. arXiv:2406.17557. https://arxiv.org/abs/2406.17557
  • Xie, S. M., Pham, H., Dong, X., et al. (2023). DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS 2023. arXiv:2305.10429. https://arxiv.org/abs/2305.10429
  • Aryabumi, V., Su, Y., Ma, R., et al. (2024). To Code, or Not To Code? Exploring Impact of Code in Pre-training. arXiv:2408.10914. https://arxiv.org/abs/2408.10914

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

Why did decoder-only causal language modeling become the dominant pretraining objective for general-purpose LLMs?

Explanation: Autoregressive generation at inference is just repeated application of the exact training objective β€” no bidirectional-context or encoder/decoder mismatch to bridge, unlike masked LM or span corruption.

What two distinct harms does Lee et al. attribute to duplicated text in pretraining corpora?

Explanation: Duplicated passages both make models more likely to regurgitate training text verbatim and waste training steps on text that teaches the model nothing new.

What did RefinedWeb and FineWeb argue, and what evidence supported their position?

Explanation: Both projects showed that the "curated corpus is necessary" assumption reflected insufficiently rigorous web-data filtering, not a fundamental limitation of web data itself.

Which architectural property does masked language modeling (as used by BERT) specifically require?

Explanation: Masked LM's entire point is to predict a corrupted token from both left and right context β€” that requires unmasked, fully bidirectional self-attention, i.e., an encoder-only stack.

A pretraining corpus contains 400 billion tokens. Deduplication removes 20% of them as exact or near-duplicate text. How many billion tokens remain after deduplication?

Explanation: 400 Γ— (1 βˆ’ 0.20) = 400 Γ— 0.80 = 320 billion tokens.

Suppose 12% of a 250-billion-token training corpus consists of duplicated passages. How many billion tokens of training compute are effectively "wasted" re-training on text the model has already seen?

Explanation: 250 Γ— 0.12 = 30 billion tokens spent on text carrying no new signal.

A 300-token input document is used for masked language modeling. Following BERT's convention of masking 15% of tokens, how many tokens are masked?

Explanation: 300 Γ— 0.15 = 45 tokens.

A raw web crawl of 2 trillion tokens is quality-filtered down to 600 billion tokens, and deduplication then removes a further 10% of those. How many billion tokens remain in the final training corpus?

Explanation: 600 Γ— (1 βˆ’ 0.10) = 600 Γ— 0.90 = 540 billion tokens.

What does DoReMi do to find a better domain mixture for pretraining?

Explanation: DoReMi trains a small proxy model, uses it to identify which domains a fixed reference model under-fits relative to their sampling weight, and re-derives domain weights from that signal β€” improving on a hand-tuned mixture without training the large model more than once to find it.