The book "Foundations of Large Language Models" is released, covering six major topics including pretraining, alignment, and reasoning.

How to obtain foundation models adaptable to multiple tasks through large-scale self-supervised pretraining, and how generative large language models are trained, aligned, and handle long contexts.

• Scenario: The authors state that large language models show that knowledge of the world and language can be acquired through large-scale language modeling, enabling general-purpose models that handle diverse problems. The research approach has shifted from training specialized systems from scratch with large amounts of labeled data to pretraining foundation models, then fine-tuning, aligning, and prompting.
• Shortcomings of existing approaches: Supervised pretraining requires more labels as models become more complex. Self-training relies on seed data and pseudo-labels. Causal language modeling only looks at left context. Masked language modeling uses [MASK], which appears only during training, and predictions of each masked token are independent.
• The authors' view: Self-supervision can construct prediction or classification signals directly from unlabeled text, allowing encoders, decoders, or encoder-decoders to learn transferable representations; when next-token prediction is scaled up, many tasks can be rewritten as text generation.
• What the book does: The preface states there are six chapters, focusing on foundational concepts and related techniques rather than covering all frontier methods. Chapter 1 discusses pretraining tasks, BERT, and its downstream adaptation; Chapter 2 discusses decoder-only generative models, large-scale training, scaling laws, and long-sequence modeling. The preface also lists that the remaining four chapters will discuss prompting, alignment, reasoning, and reasoning capabilities.

The book's related discussion falls into four groups; the authors did not run these methods as their own baselines in a unified set of experiments.

Early pretraining

• Unsupervised and supervised: Bengio et al. (2006) used layer-wise reconstruction for unsupervised pretraining; Erhan et al. (2010) discussed its relationship to subsequent supervised learning. An example of supervised pretraining is training an encoder on one classification task, then swapping the head and transferring to another task.
• Self-training: Yarowsky (1995) for word sense disambiguation and Blum and Mitchell (1998) for document classification used seed classifiers to assign pseudo-labels to unlabeled data. The authors state that self-supervised pretraining does not rely on an initial model to produce discrete pseudo-labels; the signal comes directly from the text.
• Representation learning: Mikolov et al. (2013) and Pennington et al. (2014) learned word embeddings from large-scale text. He et al. (2019) and Zoph et al. (2020) trained vision backbones on ImageNet and then used them for downstream tasks.

Self-supervised language pretraining

• Encoders: BERT (Devlin et al., 2019) performs masked language modeling and next sentence prediction. Yang et al. (2019) proposed permutation language modeling to address [MASK] appearing only during training and the independent prediction of masked tokens. Clark et al. (2019) ELECTRA uses a generator to replace tokens and a discriminator to make token-level judgments; the book states that GAN-style training is harder to scale.
• Encoder-decoder: Raffel et al. (2020) T5 casts classification, question answering, translation, and even continuous scoring as text generation, using sentinel tokens to represent masked spans. Lewis et al. (2020) BART compares masking, deletion, span masking, sentence permutation, and document rotation. Song et al. (2019) and Joshi et al. (2020) correspond to MASS-style and contiguous span masking, respectively.
• Multilingual: Lample and Conneau (2019) XLM performs translation language modeling on parallel sentences. Conneau et al. (2020) discuss that as the number of languages grows, larger models and larger shared vocabularies are needed, and describe the curse of multilinguality under fixed capacity.

Generative models, alignment, and scaling

• Language modeling lineage: Shannon (1951) estimated the predictability of English; n-grams appear in Jurafsky and Martin (2008) and others. Bengio et al. (2003) used feedforward networks to estimate n-gram probabilities and learn word embeddings. The GPT series by Radford et al. and Brown et al. are the book's main examples of the decoder-only approach.
• Scaling and alignment: Hestness et al. (2017) described test error curves in terms of data scale. Kaplan et al. (2020) and Hoffmann et al. (2022) gave power laws relating loss to parameter count and data volume. Wei et al. (2022b) discussed new capabilities emerging once scale increases past a certain point. RLHF is cited from Christiano et al. (2017), Stiennon et al. (2020), and Ouyang et al. (2022). Chain of thought is cited from Wei et al. (2022c) and Kojima et al. (2022).
• The authors' caveat on scaling curves: The authors state there is no unified scaling function; forms containing only power laws also cannot describe curves with inflection points such as double descent.

Baselines, benchmarks, and the book's positioning

• Baselines and data: The book has no experimental baselines of its own. Table 1.1 only compares the input-output of pretraining tasks and the applicable architectures. Tables 2.2 and 2.3 summarize the structures and pretraining token counts of other models. Sentiment, entailment, reading comprehension, and GSM8K appear only as task examples.
• Positioning: The authors state the book focuses on foundations rather than the full range of frontier methods, and presents the details of generative large models after the pretraining concepts.

The overall idea is to train sequence models with self-supervised tasks to obtain base models that can be fine-tuned or prompted; the generative route uses decoder-only Transformers for next-token prediction, followed by alignment and prompting.

Problem setup and two usages

• Unified model: Input is a token sequence. The network output can be a distribution over the vocabulary or a real-valued representation. Pretraining does not assume a single downstream task.
• Two model types: Sequence encoding models turn a sequence into a sequence of vectors, followed by a classification or regression head. Sequence generation models generate token by token conditioned on context.
• Adaptation: Encoding models are often fine-tuned, either training only the new head or updating the encoder as well. Generation models can be fine-tuned directly or prompted to frame the task as completion. The authors state that large models still need tuning to follow instructions reliably, and alignment with human values also depends on fine-tuning.
• Sequence probability: A next-token model writes sequence probability as Pr(x₀,…,x_m)=∏_i=0^mPr(x_i|x₀,…,x_i−1). The meaning is the product of left-to-right conditional probabilities; the book sets that conditional probability to 1 when i=0.

Self-supervised pretraining tasks

• Causal and prefix language modeling: Decoders see only the left context, trained with cross-entropy or maximum likelihood. Prefix language modeling has an encoder read the prefix once, then a decoder continues.
• Masked language modeling: Randomly mask some tokens, computing reconstruction loss only at masked positions. Can predict only masked positions, or have a decoder autoregressively reconstruct the whole sentence, i.e., denoising autoencoding.
• Permutation language modeling: Input order is unchanged; only prediction order changes, implemented with attention masks. The authors state this allows some tokens to see both left and right context and avoids a dedicated [MASK].
• Discriminative and corruption: Next sentence prediction judges whether the second sentence follows the first. ELECTRA uses a small generator to replace tokens, and a discriminator does token-level binary classification; the generator is discarded after training. BART corruptions include token masking, deletion, span masking, sentence permutation, and document rotation. T5 uses sentinel tokens to represent contiguous masked spans, shortening training sequences.

BERT and downstream adaptation

• Loss: Standard BERT is a bidirectional Transformer encoder. Total loss is Loss_BERT=Loss_mlm+Loss_nsp, the sum of masked language modeling and next sentence prediction.
• Masking details: Each sequence typically selects 15% of tokens; of those, 80% are replaced with [MASK], 10% with random tokens, and 10% left unchanged. The authors state that keeping unchanged tokens mitigates the discrepancy between pretraining with [MASK] and fine-tuning without it.
• Input and scale: Each position's embedding is the sum of token, position, and segment embeddings. BERTbase is d=768, L=12, nhead=12, 110M parameters; BERTlarge is d=1024, L=24, nhead=16, 340M parameters. The book states that a common practice is to train many steps on shorter sequences, then continue on full-length sequences.
• Usage: Remove the pretraining Softmax, attach a task head, and fine-tune on labeled data. The book lists single-text classification, text-pair classification, regression, sequence labeling, span prediction, and using BERT as the encoder in an encoder–decoder. Sequence labeling examples use BIO; reading comprehension predicts span start and end for each context position.

Generation, instruction fine-tuning, and RLHF

• Structure: Decoder-only Transformer. Embeddings are token embeddings plus position embeddings. Each layer has self-attention and FFN, with causal masking in self-attention. The output layer applies Softmax to the last layer's representation. At inference, take the next token and append it to the context.
• Training: Maximize sequence log-likelihood on the dataset. The authors state that as models grow, training becomes less stable, often requiring changes to architecture, parallelism, and initialization.
• Instruction fine-tuning: Samples are split into input and output segments; loss is computed only on the output segment. The authors state that fine-tuning data is typically orders of magnitude smaller than pretraining data and need not cover all downstream tasks; some degree of zero-shot instruction following appears after fine-tuning with relatively few labels.
• RLHF: The authors place it after pretraining and supervised fine-tuning. The process starts with an initial policy, samples multiple outputs for the same input and has humans rank them, trains a reward model with a ranking loss, then updates the policy under the reward. The book mentions PPO as an example. The reward model shares architecture with the target model but is smaller, scoring on the top-layer representation of a special symbol at the end of the sequence. The authors state that human preferences are often hard to demonstrate directly but easy to compare, so preference learning is not written as ordinary score regression. Alignment goals include following instructions, as well as being unbiased, truthful, and harmless; the authors also write safety as remaining controllable under misuse or adversarial conditions.

Scaled training and long sequences

• Data: The book relays Raffel et al., that training on unfiltered data can be harmful; and relays Penedo et al., that heuristic processing can discard 90% of web-crawled data. The book states that adding code is related not only to programming ability but also to reasoning that requires chain of thought. Multilingual performance depends on the quantity and quality of data in each language. On privacy, the authors state that it is difficult in practice to completely remove sensitive data.
• Architecture changes: Large models often use pre-norm. RMSNorm is used in the Llama series. FFN activations include ReLU, GeLU (BERT, GPT-3, BLOOM), GeGLU (Gemma), and SwiGLU (PaLM, Llama). The book relays Chowdhery et al.: removing bias terms from affine transformations helps training stability; Llama and Gemma adopt this practice.
• Parallelism: Data parallelism aggregates mini-batch gradients; naive model parallelism splits layers across devices, but devices are idle most of the time; tensor parallelism splits large matrix multiplications; pipeline parallelism overlaps computation with micro-batches. The authors state that communication, synchronization, failures, and non-associativity of low-precision accumulation all affect large-scale training.
• Long context: Standard self-attention time and memory grow quadratically with length, and KV cache grows linearly. The book discusses sparse attention, linear attention, recurrent models, fixed-size memory, k-nearest-neighbor retrieval and RAG, and MQA, GQA, and cross-layer sharing. Learnable absolute positions struggle to represent positions unseen in training. Sinusoidal encodings can formally extend, but the authors state that performance degrades when sequences are extended a lot. T5 bias buckets query–key offsets: the first half is one-to-one, the second half widens logarithmically.

Experimental Setup

• Authors' experiments: Chapter 1 and the Chapter 2 discussions on training, alignment, and long sequences contain no experiments where the authors trained or evaluated models themselves, nor any model reviews, thresholds, repetition counts, or human agreement.
• What is paraphrased: Table 2.2 summarizes the parameter counts, depths, widths, and head counts of GPT-1, GPT-2, GPT-3, Llama2, Llama3/3.1, Gemma2, Qwen2.5, DeepSeek-V3, Falcon, and Mistral. Table 2.3 summarizes the pretraining token counts and data categories for 7 models.
• Task names: Grammaticality, sentiment, semantic equivalence, entailment, commonsense inference, question-answering inference, reading comprehension, NER, machine translation, and GSM8K are only used as adaptation or prompting examples; the book gives no sample sizes or scores of its own.
• Metrics: The discussion concerns cross-entropy, log-likelihood, ranking loss, and test loss, not accuracy computed by the authors on some benchmark.
• BERT: Section 1.3.1.2 only gives architectural numbers, not downstream evaluation results.
• Comparisons: There are no baseline systems set by the authors. Table 1.1 compares the input-output formats of pretraining tasks.

Main Results

The book contains no main results from the authors' own experiments. The table below is the pretraining scale as paraphrased from Table 2.3.

Model	Token count	Main data categories (per Table 2.3)
GPT-3-175B	0.3T	Webpages, Books, Wikipedia
Falcon-180B	3.5T	Webpages, Books, Conversations, Code, Technical Articles
PaLM-540B	0.78T	Webpages, Books, Conversations, Code, Wikipedia, News
Gemma-7B	6T	Webpages, Mathematics, Code
Llama3.1-405B	15.6T	General Text, Mathematics, Reasoning Data, Code, Multilingual Text
DeepSeek-V3	14.8T	Multilingual Text, Mathematics, Code
Qwen3-235B-A22B	36T	Multilingual Text, Books, Code, Reasoning Data, Synthetic Data

• The table note states that token counts depend on the tokenizer and may include repeated sampling of training data.
• The authors state that a larger training set does not imply better training results; quality, diversity, bias, and privacy all limit the usable data.
• These numbers are the book's summary of existing models and should not be read as the authors reproducing experiments. The Gemma2 specification in Table 2.2 and the Gemma-7B in Table 2.3 are not the same row and should not be merged.

Paraphrased Architectural Scales and Scaling Laws

• Table 2.2: GPT-3 is 175B, depth 96, width 12288, heads 96/96. Llama3/3.1's 405B has depth 126, width 16384, heads 128/8. DeepSeek-V3 is 671B, depth 61, width 7168, heads 128/128. In a/b, a is the number of query heads and b is the number of key and value heads.
• Kaplan et al. (2020): The book writes L(N)=(N/8.8×10^13)^(-0.076), L(D)=(D/5.4×10^13)^(-0.095). The authors state that after an initial phase, loss decreases as a power law with model scale or data scale.
• Hoffmann et al. (2022): The Chinchilla formula is written as L(N,D)=406.4/N^0.34+410.7/D^0.28+1.69. The authors state that lower test loss does not always correspond to better performance on all downstream tasks, and the scaling relationships for different tasks may differ in practice.

Exceptions Noted in the Book

• Masked language modeling: Training uses [MASK], inference does not; predictions of masked tokens ignore dependencies among each other. Permutation language modeling is described as addressing these two points, and the book gives no comparative scores.
• Positional encoding: Absolute positional embeddings learned only at training positions cannot represent unseen positions. Sinusoidal encodings can be extended in form, but the authors state they do not work well when sequences are extended by a lot.
• Multilinguality: The book paraphrases Conneau et al., stating that with fixed capacity, too many languages lead to a curse of multilinguality, and high-resource languages decline in performance. No specific percentage points are given.
• Training systems: In naive model parallelism, devices are idle most of the time; low-precision gradient accumulation may produce numerical differences because floating-point addition is not associative. The book gives no readings on speed or convergence curves.

Other Ablations and Analyses

• Masking setup: 15% of selected tokens are chosen, then split 80%, 10%, 10% into masking, random replacement, and keeping unchanged (Section 1.3.1.1); this is a training setup, not an ablation score.
• RoBERTa: The book paraphrases Liu et al., stating that with more data and compute, BERT can be improved without changing the architecture; removing next-sentence prediction does not degrade downstream performance with more training. No scores of the book's own.
• Larger BERT-style models: The book paraphrases He et al.'s 1.5 billion parameter model and Shoeybi et al.'s 3.9 billion parameter model; the latter uses hundreds of GPUs. No downstream scores.
• mBERT: The book states it is trained on text in 104 languages, and the main difference from monolingual BERT is a larger vocabulary. No cross-lingual scores.
• Data filtering: The book paraphrases Penedo et al., stating that after heuristic processing, 90% of web-crawled data can be discarded.
• Activations and biases: GeLU is used in BERT, GPT-3, and BLOOM; SwiGLU is used in PaLM and Llama; Chowdhery et al. report that removing biases helps with stability. None of these include loss differences measured by the book.

Limitations and future directions noted by the authors

• Scope of writing: The preface states that the book focuses more on foundations rather than comprehensively covering all frontier methods.
• Research stage: Section 1.5 states that truly intelligent systems are still in early stages, with a long way to go. The authors state that what remains to be explored includes efficient learning from limited data, as well as complex reasoning and planning.
• Boundary of Chapter 1: Section 1.5 states that the chapter is mainly an introduction and cannot cover all aspects of pretraining; various fine-tuning methods are left to later chapters.
• Loss and downstream tasks: Section 2.2.4 states that lower test loss does not always mean better performance on all downstream tasks. Steps such as fine-tuning and prompting affect the final results, and scaling laws for different downstream tasks may differ in practice.
• Memory capacity: Section 2.3.3.3 states that there is no general definition of memory capacity for large language models; in practice, there is a trade-off between performance and memory usage.
• The book has no separate Limitations section. The above items come from boundaries or open issues explicitly stated in the preface, Section 1.5, Section 2.2.4, and Section 2.3.3.3.

Experimental coverage

• The discussions of pretraining, training scaling, and long sequences in Chapters 1 and 2 do not include training or downstream evaluations run by the authors themselves; the paper does not report sample sizes, number of repetitions, evaluation models, or human agreement for the authors' experiments.
• Structural numbers are written up to BERTbase's 110M parameters and BERTlarge's 340M parameters, as well as mBERT's 104 languages.
• Table 2.2 lists the parameter counts, depths, widths, and Q/KV head counts of 10 model families: GPT-1, GPT-2, GPT-3, Llama2, Llama3/3.1, Gemma2, Qwen2.5, DeepSeek-V3, Falcon, and Mistral; among them, Llama2, Llama3/3.1, Gemma2, Qwen2.5, and Falcon each have multiple sizes.
• Table 2.3 reports the pretraining token counts of 7 models, including 0.3T for GPT-3-175B and 36T for Qwen3-235B-A22B, and notes that token counts depend on the tokenizer and may involve repeated sampling.
• The comparison of pretraining tasks stops at the format illustration in Table 1.1. The scaling formulas are reported from Kaplan et al. and Hoffmann et al., with no new curves fitted by the authors.

This book by Xiao and Zhu is an introductory guide to the foundations of large language models, not a new experimental report.

• Problem: The authors aim to explain how large-scale self-supervised pretraining yields foundation models, which are then adapted through fine-tuning, alignment, and prompting to handle various language tasks, rather than training specialized systems from scratch for each task.
• Pretraining: Causal language modeling, masked and permutation language modeling, discriminative training, and denoising autoencoding can be used for decoder, encoder, or encoder-decoder models, respectively. BERT sums masked language modeling and next-sentence prediction; typically 15% of tokens are selected, then split 80%, 10%, 10% into masking, random replacement, and keeping unchanged. BERTbase has 110M parameters and BERTlarge has 340M.
• Generation and alignment: A decoder-only Transformer maximizes the log-likelihood of the next token. Instruction fine-tuning computes loss only on output segments. RLHF first collects output rankings, then trains a reward model and updates the policy. The authors state the goals include accurately following instructions, as well as being unbiased, truthful, and harmless. The prompting section uses in-context learning and chain of thought to explain how to accomplish new tasks without changing parameters.
• Scale and long context: The pretraining scales paraphrased from Table 2.3 include 0.3T tokens for GPT-3-175B and 36T for Qwen3-235B-A22B. The book paraphrases the power-law formulas of Kaplan and Hoffmann and states that lower loss does not always correspond to better performance on all downstream tasks. The long-sequence section discusses sparse and linear attention, fixed memory, retrieval augmentation, and MQA, GQA, and positional extrapolation or interpolation.
• Authors' conclusions: The authors believe self-supervised pretraining has changed how NLP research is conducted, but truly intelligent systems are still in their early stages. The authors state that efficient learning from limited data, as well as complex reasoning and planning, remain open problems. The preface states the book focuses on foundations and does not cover all frontier methods.