Chapter 5.3 β€” Multimodal LLMs

Contents

  1. From external text to a different kind of signal entirely
  2. Why images don't fit the Transformer's native assumptions
  3. CLIP: aligning images and text without generation at all
  4. Flamingo: bridging frozen encoders with trained cross-attention
  5. LLaVA: projecting images into the token stream
  6. The convergence toward unified, tokenize-everything architectures
  7. Interview angle
  8. Self-check questions
  9. Sources

1. From external text to a different kind of signal entirely

Chapter 5.2 solved a specific version of the "the model doesn't know that" problem by pulling in external text at inference time. But the fix worked because text is exactly the kind of object a Transformer already knows how to consume: a sequence of discrete tokens, embedded and processed like everything else the model has ever seen. Multimodal LLMs confront a harder version of the same integration problem β€” not "how do I bring in more of the same kind of signal," but "how do I bring in a signal that isn't naturally discrete or sequential at all." Images are continuous, high-dimensional, spatially structured grids of pixel values; there is no obvious "next pixel to predict" analogous to next-token prediction, and no natural vocabulary of image "tokens" the way there is a vocabulary of subword text tokens. This chapter traces how the field solved that integration problem, through three architecturally distinct approaches that converge, by the end, on a strikingly similar answer.

2. Why images don't fit the Transformer's native assumptions

Everything built up so far in this book β€” the chain-rule factorization of Chapter 1.1, the attention mechanism, the pretraining objective β€” assumes a sequence of discrete symbols drawn from a fixed, finite vocabulary, each one embedded into a vector via a lookup table. An image, in its raw form, is none of these things: it's a continuous-valued tensor of pixel intensities, with no natural discretization and no natural linear ordering (a pixel's meaning depends on its neighbors in two dimensions, not on some canonical left-to-right reading order the way text does). Any attempt to feed images into a Transformer-based language model therefore has to solve two problems at once: how to turn a continuous, 2D signal into something that looks like a sequence of vectors a Transformer can attend over, and how to make those vectors live in a representational space the language model's existing machinery can actually make use of, given that the language model was pretrained purely on text and has no innate notion of what a patch of pixels "means."

The three approaches covered in this chapter β€” CLIP, Flamingo, and LLaVA β€” represent three different points on a spectrum of answers to that second problem, ranging from "don't even try to generate from a language model, just align representations" (CLIP) to "keep both modalities' pretrained backbones frozen and train a bridge between them" (Flamingo) to "just project image features into the same space as text token embeddings and let an off-the-shelf decoder-only LLM treat them as more tokens" (LLaVA).

3. CLIP: aligning images and text without generation at all

Radford et al. (2021) introduced CLIP (Contrastive Language-Image Pretraining), which sidesteps the generation problem entirely by not trying to generate anything. CLIP trains two separate encoders β€” an image encoder (a Vision Transformer or a ResNet-style convolutional network) and a text encoder (a Transformer) β€” jointly, using a contrastive objective structurally similar to the one behind Dense Passage Retrieval in Chapter 5.2: given a large batch of image-caption pairs scraped from the web, encode every image and every caption into a shared embedding space, and train the two encoders so that the embedding of an image is close (by cosine similarity) to the embedding of its true matching caption, and far from the embeddings of the other captions in the batch. Concretely, for a batch of $N$ image-text pairs, CLIP computes an $N \times N$ similarity matrix between all images and all texts, and applies a symmetric cross-entropy loss that treats each image's true caption as the positive class among the $N$ candidate captions (and vice versa for each caption's true image) β€” a large in-batch contrastive objective that scales naturally with batch size, since larger batches supply harder and more numerous negatives.

The remarkable empirical result is that the resulting image encoder, despite never being trained to caption, describe, or generate anything, produces image representations that transfer extraordinarily well to downstream tasks it was never explicitly trained on β€” zero-shot image classification (by comparing an image's embedding to the embeddings of candidate class-name captions like "a photo of a dog"), retrieval, and, most importantly for this chapter, serving as a general-purpose visual feature extractor for other systems to build on. CLIP is not itself a multimodal generative LLM β€” it has no mechanism for producing text conditioned on an image beyond nearest-neighbor comparison against a fixed set of candidate captions β€” but it became foundational infrastructure for the field precisely because it demonstrated that a well-aligned, semantically meaningful shared embedding space between vision and language could be learned purely from noisy, web-scale image-caption pairs, without manual annotation. Both of the two generative architectures that follow depend on a CLIP-style vision encoder (or one trained the same way) as a starting point.

4. Flamingo: bridging frozen encoders with trained cross-attention

Alayrac et al. (2022) tackled the harder problem CLIP sidesteps: building a model that can actually generate free-form text conditioned on images, ideally with the same few-shot in-context learning behavior GPT-3-style language models showed for text (Part IV's discussion of in-context learning). Flamingo's approach is notable for what it deliberately avoids doing: rather than training a new multimodal model from scratch, which would require redoing the enormous investment already sunk into pretraining a strong vision encoder and a strong language model separately, Flamingo keeps both a pretrained vision encoder and a pretrained language model entirely frozen, and trains only a comparatively small set of new components that bridge them.

The bridge has two pieces. First, a "Perceiver Resampler" module takes the vision encoder's output β€” which can vary in spatial size depending on the input image or video β€” and compresses it into a small, fixed number of visual tokens via learned cross-attention, so that downstream processing doesn't have to deal with variable-length, potentially very large visual feature maps. Second, and this is the architecturally distinctive part, new cross-attention layers are interleaved between the existing (frozen) layers of the language model, gated so that at initialization they contribute nothing and the language model behaves exactly as it did before β€” a stability trick reminiscent of the residual and gating conventions covered in earlier architecture chapters. These new cross-attention layers let each language-model layer attend over the visual tokens produced by the resampler, conditioning text generation on visual input while never touching the original language model's own weights.

The practical payoff is that Flamingo can perform few-shot multimodal in-context learning: given a prompt containing a handful of interleaved image-text examples followed by a new image, it can produce an appropriate text continuation without any gradient updates or task-specific fine-tuning, extending the in-context learning phenomenon from pure text into the multimodal setting. The cost of this approach is architectural complexity β€” new cross-attention layers have to be trained and carefully interleaved into an existing, frozen language model's forward pass, which is a nontrivial engineering undertaking compared to simply concatenating tokens.

5. LLaVA: projecting images into the token stream

Liu et al. (2023) proposed a strikingly simpler alternative that turned out to be highly influential precisely because of that simplicity: LLaVA (Large Language and Vision Assistant) uses a frozen pretrained vision encoder (a CLIP-style encoder) to extract visual features from an image, and then trains a lightweight projection β€” in the simplest version, a single linear layer, later extended to a small multilayer perceptron β€” that maps those visual features into the same embedding-dimensional space as the language model's text token embeddings. The projected visual features are then simply inserted into the input sequence alongside the text token embeddings, and an off-the-shelf, entirely standard decoder-only LLM processes the combined sequence exactly as it would process a sequence of pure text tokens, with no architectural modification to the language model itself β€” no new cross-attention layers, no gating, nothing bespoke about how attention works internally.

This works because the projection layer's job is narrowly defined: learn a mapping from "CLIP's visual feature space" to "this specific language model's token embedding space," such that the projected vectors land somewhere the language model's existing attention and feed-forward layers can meaningfully process, the same way a normal text token embedding would be processed. Training LLaVA involves two stages: first, pretraining just the projection layer (keeping both the vision encoder and the language model frozen) on image-caption pairs, so the projection learns to produce embeddings the language model can interpret at all; second, "visual instruction tuning" β€” fine-tuning (typically the projection layer and often the language model as well) on a dataset of instruction-following examples that involve images, generated in part by prompting a strong text-only LLM to produce plausible instruction-response pairs conditioned on image descriptions, so the resulting model learns to follow open-ended instructions about image content rather than merely captioning.

The comparison with Flamingo is instructive for interview purposes: LLaVA's projection-based approach is architecturally much simpler and cheaper to train β€” a linear or small-MLP projection plus instruction tuning, versus new cross-attention layers threaded through every layer of a frozen LLM β€” while achieving strong performance on visual instruction-following benchmarks, and it demonstrated that a decoder-only LLM's existing attention machinery is flexible enough to handle visual "tokens" it was never originally trained on, provided those tokens are mapped into a compatible embedding space first.

6. The convergence toward unified, tokenize-everything architectures

Looking across CLIP, Flamingo, and LLaVA, there's a clear trajectory: from not generating at all (CLIP), to generating via new bespoke architecture bridging two frozen backbones (Flamingo), to generating by simply treating projected visual features as more tokens fed into an otherwise-unmodified decoder-only LLM (LLaVA). The field has continued moving in the direction LLaVA points toward: rather than building modality-specific architectural machinery β€” separate cross-attention pathways, separate encoders bolted on with custom integration logic β€” the emerging convergence is toward architectures where every modality (images, audio, and beyond) is tokenized, by some encoder or discrete tokenizer appropriate to that modality, into a sequence of vectors or discrete codes that live in a shared representational space, and then handled by exactly the same decoder-only Transformer machinery that processes text. Under this view, a "vision encoder" becomes conceptually similar to a "tokenizer" from Part IV's tokenization discussion: a modality-specific preprocessing step whose entire job is to turn raw input into something that looks, to the core Transformer, just like more tokens in the stream. This unification is attractive for the same reason decoder-only architectures won out for text in the first place (a theme from earlier architecture chapters): one general-purpose mechanism, attention over a token sequence, scales better and is easier to reason about than a proliferation of modality-specific bespoke components. OpenAI's GPT-4o is the clearest production instance of this convergence actually shipped: a single model that natively processes and generates text, images, and audio, rather than a text-only LLM bolted to separate specialist models for each other modality behind an orchestration layer β€” the "o" is short for "omni," and the whole point of the name is that perception and generation across modalities live in one model rather than a pipeline of several.

Everything covered so far in this chapter is about a model perceiving images, not producing them, and image generation turns out to split into two genuinely different paradigms worth telling apart. Diffusion models β€” the mechanism behind DALL-E 2/3, Stable Diffusion, and Imagen β€” generate an image by starting from pure noise and iteratively denoising it over many steps, guided at each step by the conditioning text, gradually turning static into a coherent image rather than producing it in one shot. Autoregressive image generation instead extends this chapter's tokenize-everything theme to generation, not just understanding: an image is first converted into a sequence of discrete tokens from a learned visual vocabulary (via a discrete tokenizer like VQ-VAE or VQGAN, conceptually the image analog of Chapter 4.1's text tokenizers), and then a decoder-only Transformer generates that token sequence one token at a time, exactly the same next-token-prediction mechanism used for text, as in Google's Parti and β€” combining generation and understanding in one unified token stream β€” Meta's Chameleon and GPT-4o's own image generation. Diffusion currently dominates on raw image fidelity for pure text-to-image generation; the autoregressive route trades some of that fidelity for the architectural benefit this whole section has been building toward β€” one shared mechanism handling perception and generation, across every modality, rather than a diffusion model bolted on as a separate specialist component.

Multimodal integration solves the problem of what kind of input a model can perceive. It says nothing about what the model can do with that perception once conditioning is in place β€” in particular, whether the model can go beyond describing or reasoning about what it perceives and actually act on the world, using external tools to execute code, query live data, or take real actions. That is where the next chapter turns.

7. Interview angle

Q: Why can't you just feed raw pixels into a language model's token embedding lookup the way you feed in text tokens? A strong answer notes that text tokens come from a small, fixed, discrete vocabulary with a learned embedding table, whereas raw pixels are continuous-valued and have no natural discretization or canonical sequential order; any integration approach has to first convert the image into a sequence of vectors (via an encoder) that can be treated the way token embeddings are treated, which is exactly the design problem CLIP, Flamingo, and LLaVA each solve differently.

Q: CLIP isn't a generative multimodal model β€” so why is it foundational to models that are? A strong answer explains that CLIP's contrastive training produces an image encoder whose output lives in a semantically meaningful, text-aligned embedding space, purely from web-scale image-caption pairs with no manual annotation; this pretrained encoder becomes the visual feature extractor that Flamingo and LLaVA both build on top of, meaning CLIP's contribution is representational infrastructure rather than generation itself.

Q: Compare Flamingo's and LLaVA's approach to bridging vision and language. What's the practical tradeoff? A strong answer covers: Flamingo keeps both backbones frozen and trains new interleaved cross-attention layers plus a Perceiver Resampler to compress variable-size visual features into a fixed set of tokens, enabling few-shot multimodal in-context learning but requiring nontrivial new architecture. LLaVA projects visual features via a lightweight linear or MLP layer directly into the language model's token embedding space and feeds them through an entirely unmodified decoder-only LLM, which is far simpler to implement and train, at the cost of relying on the projection alone to bridge two very differently-trained representational spaces.

Q: What does "visual instruction tuning" mean, and why is it a separate stage from the initial vision-language alignment pretraining in LLaVA? A strong answer distinguishes an initial stage where only the projection layer is trained (frozen vision encoder and frozen LLM) on image-caption data, purely to teach the projection to produce embeddings the LLM can interpret at all, from a second stage that fine-tunes on instruction-following data involving images so the model learns to answer open-ended questions and follow instructions about image content, not merely produce captions.

Q: Where do you see multimodal LLM architecture heading, based on the CLIP-to-Flamingo-to-LLaVA progression? A strong answer identifies the trend toward unified, "tokenize everything" decoder-only architectures, where a modality-specific encoder or tokenizer's only job is to produce a sequence of vectors or discrete codes compatible with the language model's existing token embedding space, avoiding bespoke cross-modal architecture like Flamingo's interleaved cross-attention in favor of reusing one general attention mechanism across modalities.

8. Self-check questions

  1. What two properties of raw images make them incompatible with a Transformer's native assumption of a discrete token sequence?
  2. Describe CLIP's contrastive training objective, including what the $N \times N$ similarity matrix represents for a batch of $N$ image-text pairs.
  3. Why is CLIP described as "foundational infrastructure" rather than a multimodal generative model in its own right?
  4. What two new components does Flamingo add on top of frozen pretrained vision and language backbones, and what does each do?
  5. How does LLaVA's projection layer make an entirely unmodified decoder-only LLM able to process visual input?
  6. What are the two training stages of LLaVA, and what does each stage's data and frozen/unfrozen configuration accomplish?
  7. In what sense do CLIP, Flamingo, and LLaVA represent points along a single trajectory, and what does that trajectory suggest about where multimodal architecture is converging?

9. Sources

  • Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML 2021. arXiv:2103.00020. https://arxiv.org/abs/2103.00020
  • Alayrac, J.-B., Donahue, J., Luc, P., et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022. arXiv:2204.14198. https://arxiv.org/abs/2204.14198
  • Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual Instruction Tuning (LLaVA). NeurIPS 2023. arXiv:2304.08485. https://arxiv.org/abs/2304.08485
  • OpenAI (2024). GPT-4o System Card. https://openai.com/index/gpt-4o-system-card/
  • Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239. https://arxiv.org/abs/2006.11239
  • Yu, J., Xu, Y., Koh, J. Y., et al. (2022, Google). Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (Parti). arXiv:2206.10789. https://arxiv.org/abs/2206.10789
  • Chameleon Team (2024, Meta AI). Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv:2405.09818. https://arxiv.org/abs/2405.09818

Self-check quiz

Test your understanding of this chapter with a short quiz β€” a mix of concept questions and quick calculations.

Why can't raw pixels be fed into a language model's token embedding lookup the way text tokens are?

Explanation: A pixel's meaning depends on 2D spatial neighbors, not a canonical left-to-right order β€” any integration approach must first convert the image into a sequence of vectors an encoder produces.

What is CLIP's training objective?

Explanation: CLIP computes an NΓ—N similarity matrix over a batch and applies a symmetric cross-entropy loss treating each image's true caption as the positive class among N candidates.

What are the two new components Flamingo adds on top of frozen pretrained vision and language backbones?

Explanation: The gating ensures the new cross-attention layers contribute nothing at initialization, so the frozen language model's original behavior is preserved as a stable starting point.

How does LLaVA enable an entirely unmodified decoder-only LLM to process visual input?

Explanation: No architectural modification to the LLM is needed β€” the projected visual features are simply inserted alongside text token embeddings in the input sequence.

CLIP trains on a batch of $N = 256$ image-text pairs, computing an $N \times N$ similarity matrix. How many entries does this matrix contain?

Explanation: 256 Γ— 256 = 65,536 entries.

In a CLIP training batch of $N = 8$ image-text pairs, how many negative (non-matching) caption candidates does each image have to be contrasted against?

Explanation: Each image has 1 true matching caption and Nβˆ’1 = 7 negative captions in the batch.

A Vision Transformer splits a 224Γ—224 pixel image into non-overlapping 16Γ—16 patches. How many patches (tokens) does this produce?

Explanation: (224/16) Γ— (224/16) = 14 Γ— 14 = 196 patches.

An image produces 576 patch tokens from the vision encoder. Flamingo's Perceiver Resampler compresses this to a fixed 64 tokens. What is the compression ratio (original tokens / compressed tokens)?

Explanation: 576 / 64 = 9 β€” a 9x reduction, giving downstream processing a fixed, manageable number of visual tokens regardless of input size.