OLMo: Accelerating Language Model Science

The training data, architecture, and development details of the strongest language models are blocked behind proprietary interfaces, making it difficult for the research community to scientifically study their capabilities, weaknesses, biases, and risks.

• Context and importance: The authors state that language models have commercial value after large-scale pretraining and alignment labeling, but the strongest models are becoming increasingly closed. They believe that full access to open models is crucial for studying the strengths, weaknesses, biases, and risks of these models.
• Current openness varies: The authors give examples: Mixtral 8x7B provides only weights and a brief report, LLaMA includes adaptation training instructions, MPT provides dataset distributions but not the data itself, Falcon partially releases pretraining data; Pythia and BLOOM release training code, checkpoints, and data. The authors believe most prior work only releases weights and inference code.
• Core claim: The authors argue that a large and diverse family of open models is the best hope for understanding language models and improving their usefulness. They also point to open questions such as how data affects capabilities, design and hyperparameter choices, and the impact of optimization methods on training.
• What this paper proposes: The authors release OLMo, a competitive open language model and its framework, including 1B and 7B weights, training on at least 2T tokens, the open pretraining data Dolma, training and evaluation code, multiple checkpoints across hardware, training logs, and instruction and preference adaptation using Open Instruct. Code and weights are under Apache 2.0.

The authors position the work mainly by degree of openness and by the architecture, data, and evaluation work it draws on, rather than proposing new attacks or defenses.

Model releases with varying degrees of openness

• Pythia (Biderman et al., 2023) and BLOOM (BigScience et al., 2022): the authors state these were the most open models before, releasing training code, checkpoints, data, and more. Figure 2 uses intermediate Pythia-6.9B checkpoints for perplexity comparison.
• Falcon (Almazrouei et al., 2023), LLaMA / Llama 2 (Touvron et al., 2023a,b), MPT (MosaicML NLP Team, 2023), Mixtral 8x7B (Jiang et al., 2024): the authors describe these respectively as partially releasing pretraining data, giving detailed adaptation training instructions, releasing dataset distributions but not the data itself, and releasing only weights and a short report.
• LLM360 (Liu et al., 2023): the authors state this work has similar goals and they are not the only team doing this; they also state OLMo narrows the gap between its models and capabilities like Llama 2.

Models drawn on for architecture and optimization

• Decoder-only Transformer (Vaswani et al., 2017) and PaLM, LLaMA, OpenLM, Falcon: OLMo builds on these, removing biases, using non-parametric layer norm, SwiGLU, RoPE, and a vocabulary based on GPT-NeoX-20B's BPE with added PII-masking tokens.
• AdamW (Loshchilov and Hutter, 2019), ZeRO (Rajbhandari et al., 2019), and FSDP (Zhao et al., 2023): training uses AdamW, with FSDP sharding weights and optimizer states; mixed precision is bfloat16 with softmax kept in full precision.

Data, Adaptation, and Evaluation

• Dolma (Soldaini et al., 2024): a multi-source pretraining corpus whose pipeline includes language filtering, quality filtering, content filtering, deduplication, multi-source mixing, and tokenization. The authors place finer construction and curation experiments in the Dolma report and use WIMBD (Elazar et al., 2024) for data analysis.
• Instruction and preference learning: the authors cite instruction tuning (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022), further training from human preferences (Ouyang et al., 2022), and DPO (Rafailov et al., 2023). Adaptation follows TÜLU's data and training setup (Ivison et al., 2023; Wang et al., 2023).
• Catwalk (Groeneveld et al., 2023) and Paloma (Magnusson et al., 2023): used for downstream evaluation and perplexity-based evaluation, respectively. Paloma contains 585 text domains; the zero-shot protocol borrows rank classification from Brown et al. (2020), and the task set is close to LLaMA's commonsense reasoning set.

Baseline methods are public checkpoints of similar scale: StableLM 1.6B, Pythia 1B, TinyLlama 1.1B at the 1B tier; Falcon-7B, LLaMA 7B, Llama 2 7B, MPT-7B, Pythia 6.9B, RPJ-INCITE-7B at the 7B tier. Benchmarks are 8 core zero-shot tasks (arc_challenge, arc_easy, boolq, hellaswag, openbookqa, piqa, sciq, winogrande) plus Paloma bits per byte. Adaptation comparisons include the official instruction versions of these models and TÜLU 2 and TÜLU 2+DPO, which also use the TÜLU mix.

The authors position the paper as: releasing not just weights and inference code, but also training data, training and evaluation code, intermediate checkpoints, and training logs, with the hope of narrowing the gap to models like Llama 2.

The authors release the OLMo framework: models, pretraining data Dolma, training and inference code, intermediate checkpoints, training logs, Catwalk / Paloma-based evaluation, and Open Instruct-based adaptation.

Models and Architecture

• Scale: decoder-only Transformer. Table 1 gives 1B (L=16, D=2048, H=16, 2T tokens, peak learning rate 4.0E-4, warmup 2000 steps, tied weights, batch about 4M) and 7B (L=32, D=4086, H=32, 2.46T tokens, peak learning rate 3.0E-4, warmup 5000 steps, untied weights, batch about 4M). Both use AdamW with betas 0.9 and 0.95 and epsilon 1.0E-5.
• Changes relative to the original Transformer: all biases removed; non-parametric layer norm without affine transformation, which the authors consider the most stable and fastest among the variants considered, with parametric layer norm and RMSNorm as comparisons; SwiGLU activation with hidden dimension about 8/3 d rounded to the nearest multiple of 128 (11,008 for 7B); RoPE replaces absolute positional encodings.
• Vocabulary: modified GPT-NeoX-20B BPE with added tokens for masking personally identifiable information, vocabulary 50,280; embedding matrix padded to 50,304 to be a multiple of 128. Hyperparameters are chosen by hardware throughput, with an eye to reducing the risk of loss spikes and slow divergence, and choices are ablated through in-training evaluation.

Pretraining Data Dolma

• Composition: Table 2, counted with GPT-NeoX tokenization: Common Crawl web pages 2,180B tokens, GitHub code 342B, Reddit 80B, Semantic Scholar papers 57B, Project Gutenberg 5.2B, Wikipedia 3.7B, totaling 2,668B tokens and 4,367M documents. Documents from each source remain separate in curation and in the final release.
• Pipeline: (1) language filtering, (2) quality filtering, (3) content filtering, (4) deduplication, (5) multi-source mixing, (6) tokenization. Design details are in the Dolma report. Training samples a 2T-token subset from Dolma, appends EOS at document ends, concatenates, then cuts into 2048-token instances and shuffles; shuffling is identical across runs, and the data order can be reconstructed from released artifacts.

Training Setup

• Distribution: ZeRO + PyTorch FSDP shards weights and optimizer states. The 7B uses 4096 tokens per GPU micro-batch on the hardware used. Global batch for both 1B and 7B is about 4M tokens (2048 sequences of length 2048). Mixed precision uses bfloat16 with full-precision softmax; sharded weights and optimizer states remain full precision, and gradients are reduced across GPUs in full precision.
• Learning rate: the warmup in Table 1 does not fully match the text. Section 3.2 says all scales warm up for 5000 steps (about 21B tokens), then decay linearly to one-tenth of the peak; after warmup, the total ℓ2 norm of parameter gradients is clipped to at most 1.0. The OLMo-7B checkpoint used for evaluation trains to 2.46T tokens; the authors also train 1000 more steps on Dolma, linearly annealing the learning rate to 0. Some models exceed 2T tokens in a second pass with a different shuffle order; the authors cite Muennighoff et al. (2023) and consider the effect of repeating this small fraction of data negligible.
• Hardware: up to 256 nodes on LUMI, each with 4 AMD MI250X (128GB, 800Gbps); 27 nodes on MosaicML, each with 8 NVIDIA A100 (40GB, 800Gbps). The authors state batch sizes were slightly adjusted for throughput on each side, and performance on the evaluation suite at 2T tokens is nearly identical. The released 7B variants correspond to different architectures, optimizers, and training hardware: four 7B variants and one 1B model.

Adaptation and Evaluation Protocol

• Adaptation: instruction tuning on distilled and human-written instruction data, then DPO on distilled preference data, targeting a general-purpose chat assistant, with the setup from TÜLU.
• In-loop evaluation: downstream evaluation every 1000 steps (about 4B tokens), used to decide architecture, initialization, optimizer, learning-rate schedule, and data mix.
• Offline downstream: 8 core tasks zero-shot using rank classification: candidate completions are ranked by likelihood and accuracy is reported. Normalization is chosen per dataset: arc and openbookqa use unconditional normalization, hellaswag, piqa, and winogrande use per-token, and boolq and sciq use none.
• Paloma: reports bits per byte. The authors state OLMo-7B is the largest LM with explicit perplexity decontamination: following Paloma's practice, pretraining documents containing leaked Paloma evaluation passages are removed. Aggregation uses 11 of Paloma's 18 sources, excluding non-public, marginal, or toxic text, and code data unsupported by the decontamination method.
• Adaptation evaluation: safety and chat metrics are in question 4. MMLU is 0-shot; ToxiGen uses a roberta-large toxicity classifier, generating to a stop token or at most 512 tokens, reporting the proportion of generations judged toxic; TruthfulQA uses a generation setting (818 questions, 6 in-context examples, greedy decoding), judged by two LLaMA 2-based classifiers for truthfulness and informativeness, with the primary metric % Informative and Truthful; AlpacaEval compares the model against Davinci-003 on 805 prompts using GPT-4 (alpaca_eval_gpt4), with models generating at most 2048 tokens, reporting win rate.

Experimental Setup

• Pretrained models: OLMo-1B (2T tokens) and OLMo-7B (reported at the 2.46T token checkpoint). Comparison models are StableLM 1.6B, Pythia 1B, TinyLlama 1.1B, Falcon-7B, LLaMA 7B, Llama 2 7B, MPT-7B, Pythia 6.9B, RPJ-INCITE-7B. The authors state they also release four 7B variants, and Table 3 reports only one of them, OLMo-7B.
• Downstream benchmarks: 8 core zero-shot tasks, with accuracy and 8-task average as metrics. Appendix C has additional tasks the authors consider less stable in trend; the main text does not give the numbers for that table.
• Paloma: 11 public sources and their combined bits per byte, with the x-axis as tokens seen (billions). Comparisons with intermediate checkpoints include Pythia-6.9B and RPJ-INCITE-7B.
• Adaptation: OLMo base, OLMo+SFT, OLMo+SFT+DPO, compared with MPT Chat, Falcon Instruct, RPJ-INCITE Chat, Llama-2-Chat, TÜLU 2, TÜLU 2+DPO. Metrics are MMLU 0-shot, AlpacaEval %win, ToxiGen % Toxic (lower is better), TruthfulQA %Info+True. Table 8 additionally reports GSM8k (8-shot CoT), BBH (3-shot CoT), TydiQA (1-shot), Codex-Eval Pass@10.
• Evaluation: Downstream is likelihood ranking, not generative evaluation. AlpacaEval's evaluator is GPT-4; ToxiGen's classifier is a roberta-large fine-tuned by Hartvigsen et al. (2022); TruthfulQA uses two LLaMA 2 classifiers. The paper does not report the number of repetitions.
• Decontamination: OLMo's pretraining documents were removed based on Paloma paragraph leakage. The authors follow Ivison et al. (2023) in not reporting TÜLU 2's TruthfulQA due to test set contamination; they also cite Touvron et al. (2023b), stating that Llama 2's pretraining data contains MMLU test data.

Main Results

Zero-shot accuracy from Table 3 (OLMo-7B is the 2.46T token checkpoint):

Swipe horizontally to read the table

Model	arc_c	arc_e	boolq	hellaswag	obqa	piqa	sciq	wino	avg
StableLM 1.6B	43.8	63.7	76.6	68.2	45.8	74.0	94.7	64.9	66.5
Pythia 1B	33.1	50.2	61.8	44.7	37.8	69.1	86.0	53.3	54.5
TinyLlama 1.1B	34.8	53.2	64.6	58.7	43.6	71.1	90.5	58.9	59.4
OLMo-1B	34.5	58.1	60.7	62.5	46.4	73.7	88.1	58.9	60.4
Falcon-7B	47.5	70.4	74.6	75.9	53.0	78.5	93.9	68.9	70.3
LLaMA 7B	44.5	67.9	75.4	76.2	51.2	77.2	93.9	70.5	69.6
Llama 2 7B	48.5	69.5	80.2	76.8	48.4	76.7	94.5	69.4	70.5
OLMo-7B	48.5	65.4	73.4	76.4	50.4	78.4	93.8	67.9	69.3

7B models not listed in the table: MPT-7B averages 69.8, Pythia 6.9B averages 63.0, RPJ-INCITE-7B averages 66.6 (individual scores in Table 3).

• Authors' interpretation: Overall, OLMo-7B is competitive with all comparable models. StableLM 1.6B is included in the comparison, but the authors note it is significantly larger and its training data is unknown.
• Figure 1: OLMo-7B's accuracy on 8 tasks as a function of tokens seen. All tasks show an upward trend except OBQA. The authors state that on most tasks, linearly decaying the learning rate to 0 over the last 1000 steps brings gains, with the curve turning up between the second-to-last and last steps. Readable y-axis scales in the figure include arc_c about 40–48, arc_e about 60–68, boolq about 56–76, hellaswag about 68–76, obqa about 45–51, piqa about 76–78, sciq about 90–94, winogrande about 63–66; the x-axis is Tokens Seen (billions), with ticks from 500 to 2500.
• Per-task: OLMo-7B's arc_challenge is 48.5, same as Llama 2 7B, higher than LLaMA 7B's 44.5; arc_easy is 65.4, lower than Falcon-7B's 70.4 and Llama 2 7B's 69.5; boolq is 73.4, lower than Llama 2 7B's 80.2. OLMo-1B averages 60.4, higher than Pythia 1B's 54.5 and TinyLlama 1.1B's 59.4, lower than StableLM 1.6B's 66.5.

Bits per byte on Paloma

• What was measured: Figure 2 compares OLMo-7B with Falcon-7B, LLaMA2-7B, MPT-7B, LLaMA-7B, Pythia-6.9B, RPJ-INCITE-7B, including final models (shapes) and intermediate checkpoints (dashed lines). Subplots are Sources Combined, C4, mC4, WikiText-103, PTB, RedPajama, Falcon RefinedWeb, Dolma V1.5, M2D2 S2ORC, M2D2 Wikipedia, C4 100 Domains, 100 Subreddits. The y-axis is Bits Per Byte, the x-axis is Tokens Seen (billions), with ticks at 10, 100, 1000, 10000. Y-axis ticks marked in the figure include 0.55, 0.61, 0.67, 0.74, 0.82, 0.90, 1.00, 1.11, 1.22, 1.35; the paper does not give exact final readings for each model in the main text.
• Authors' interpretation: OLMo's fit is competitive, and the training data was explicitly decontaminated against Paloma. The performance of intermediate checkpoints is affected by their position in the learning rate schedule; models with fewer steps have steeper curves and are not necessarily more sample-efficient. MPT-7B leads other models in the combined subplot. Possible factors the authors give include: MPT's non-Common Crawl data is 27%, while LLaMA is 18%, RedPajama is 12.2%, OLMo is 11.2%, and MPT uses semantic deduplication on C4.
• Per-domain: The authors state that differences in sample efficiency mainly come from whether training and evaluation distributions are similar. OLMo-7B performs well on Common Crawl-dominated evaluations (e.g., C4); the Figure 2 caption states it surpasses all other models on C4, possibly because Common Crawl makes up 88.8% of the pretraining data. On Falcon RefinedWeb, Falcon-7B, trained on that data, fits best. The authors state that on sources farther from crawled web pages, such as WikiText-103, M2D2 S2ORC, M2D2 Wikipedia, OLMo-7B's sample efficiency is lower; RedPajama is similar, as only 2 of its 7 domains come from Common Crawl, and Paloma weights domains equally within each source.

Instruction fine-tuning and DPO

Table 4 (7B instruction models; ToxiGen lower is better):

Swipe horizontally to read the table

Model	MMLU	AlpacaEval	ToxiGen	TruthfulQA
OLMo (base)	28.3	-	81.4	31.6
OLMo+SFT	47.3	57.0	14.4	41.2
OLMo+SFT+DPO	46.2	69.3	1.7	52.0
Llama-2-Chat	46.8	87.3	0.0	26.3
TÜLU 2	50.4	73.9	7.0	51.7
TÜLU 2+DPO	50.7	85.1	0.5	-

MPT Chat is 33.8 / 46.8 / 0.1 / 42.7, Falcon Instruct is 25.2 / 14.0 / 70.7 / 27.2, RPJ-INCITE Chat is 27.0 / 38.0 / 46.4 / 53.0 (order is MMLU, AlpacaEval, ToxiGen, TruthfulQA).

• Authors' interpretation: Instruction fine-tuning significantly improves OLMo-7B's performance and safety, with a large increase in MMLU, and ToxiGen and TruthfulQA further improve after DPO. The authors state that after SFT and SFT+DPO, OLMo-7B surpasses most other chat variants, attributing this to both the base model and the TÜLU mixture. There is still a gap with TÜLU 2, which the authors believe may come from Llama 2's test set contamination and the TÜLU mixture being designed mainly for Llama.
• Other tasks in Table 8: OLMo-7B base GSM8k 8.5, BBH 31.7, TydiQA 32.3, Codex-Eval Pass@10 21.4. +SFT are 15.5, 36.9, 35.2, 28.6 respectively; +SFT+DPO are 11.0, 35.8, 21.7, 27.8 respectively. After DPO, all four are lower than SFT, with TydiQA dropping from 35.2 to 21.7, below the base's 32.3. Table 8's +SFT+DPO MMLU is 46.1, Table 4's is 46.2; the paper does not explain this difference. The authors do not report TÜLU 2's TruthfulQA.
• Hardware comparison: The authors state that the LUMI and MosaicML runs at 2T tokens perform almost identically on the evaluation suite. The paper does not give per-run numbers for these two runs.

Other ablations and analyses

• Architecture choices were made through in-loop evaluation every 1000 steps; the main text does not give per-variant scores for each architecture.
• Non-parametric layer norm vs. parametric layer norm and RMSNorm: the authors state it is fastest and most stable, without giving numbers.
• The evaluated 7B checkpoint was trained on Dolma for an additional 1000 steps with learning rate decayed to 0: the authors state this improves perplexity and downstream tasks, with specific gains pointing to Appendix Table 7, not listed in the main text.
• Learning rate decay to 0 over the last 1000 steps: in Figure 1, most tasks rise in the final segment except OBQA.
• Common Crawl proportion: the authors write OLMo has 11.2% non-Common Crawl (the caption says 88.8% of pretraining data is Common Crawl), compared with LLaMA 18%, RedPajama 12.2%, MPT 27%.
• In the conclusion, the authors state that since the first version described in this paper, data and training settings have improved, e.g., MMLU increased by 24 points to 52%; this number is not in Table 3, Table 4, or Table 8.

Limitations and future directions noted by the authors

• Data language: The Data subsection of Limitations states that the work focuses on English pretraining data, and hopes the open framework will enable more languages and multilingual models. (Limitations)
• Data content: The same subsection states that data at this scale is likely to contain toxic language, personal information, and copyrighted text; the authors have tried to mitigate this, but believe there is currently no way to completely remove such content. (Limitations)
• Training failure records: The Training subsection states that, within limited space, a large number of training logs recording runs such as divergences or failures to learn were not provided; it also states that large-scale training currently lacks sufficient support from the open-source community. (Limitations)
• Adaptation data mismatch: The Adaptation subsection states that pretrained models still have bias, toxicity, and hallucination; adapted models are better at avoiding such generations, but are not perfect. They largely adopt the TÜLU mixture designed for another model family; OLMo may need a different data mixture to match its own strengths and weaknesses; the mixture relies on data distilled from multiple models, and the authors hope to reduce this reliance in the future. (Limitations)
• Evaluation does not match usage: The Evaluation subsection states that many downstream tasks do not represent users' actual interactions with language models (i.e., as chatbots); evaluation is currently very noisy, and they only include datasets that provide some signal of "which model is better," but believe there is no perfect automatic evaluation and comparisons should be viewed with caution. (Limitations)
• Future plans: The Conclusion states that the authors intend to continue supporting and expanding OLMo to include different model scales, modalities, datasets, safety measures, and evaluations. They hope to reduce reliance on distilled data and enable models in more languages. Limitations also states that the social impact exceeds what this section can cover. (Conclusion and Future Work; Limitations)

Experimental coverage

• The zero-shot main table covers 4 public checkpoints in the 1B tier and 7 in the 7B tier, across 8 core tasks; the OLMo-7B numbers correspond to the 2.46T token checkpoint (Table 3). Specific numbers for auxiliary tasks in the appendix do not appear in the provided main-text tables.
• Paloma reports bits per byte for 11 sources and the combined curve; on the OLMo side, paragraph-level leakage decontamination was performed; the main text does not list the exact endpoint values for each model (Figure 2, Section 4.2).
• Adaptation evaluation includes base, SFT, and SFT+DPO, with metrics MMLU, AlpacaEval, ToxiGen, and TruthfulQA; Table 8 additionally includes GSM8k, BBH, TydiQA, and Codex-Eval; TruthfulQA is not reported for TÜLU 2, and the authors cite test-set contamination as the reason (Table 4, Table 8, Section 4.3).
• Training hardware includes AMD MI250X on LUMI and NVIDIA A100 on MosaicML; the authors state that performance is nearly identical at 2T tokens, and the paper does not report per-run breakdown scores or the number of repetitions (Section 3.4).
• In-the-loop ablations cover architecture, initialization, optimizer, learning rate schedule, and data mixture; the main text does not report scores for each variant; Dolma's filtering, deduplication, and mixture experiments are described in the Dolma report and are not expanded here (Section 2.2, Section 2.4).

Allen Institute releases OLMo: a language model framework open from data to training to evaluation, intended to support research on language model capabilities, weaknesses, biases, and risks.

• Problem: the authors believe the strongest models are blocked behind proprietary interfaces, with insufficient data, architecture, and development details, and that releasing only weights and inference code is not enough.
• Approach: 1B and 7B decoder-only Transformers, biases removed, using non-parametric layer norm, SwiGLU, and RoPE, with a vocabulary that adds PII-masking tokens to GPT-NeoX BPE. Data comes from Dolma (Table 2 totals 2,668B tokens). AdamW + FSDP, global batch about 4M tokens, sequence length 2048. The 7B reported checkpoint trains to 2.46T tokens, then 1000 more steps annealing the learning rate to 0. Adaptation is instruction tuning on the TÜLU mix followed by DPO.
• Downstream results: in Table 3, OLMo-7B averages 69.3 on eight zero-shot tasks, Llama 2 7B is 70.5, Falcon-7B is 70.3, Pythia 6.9B is 63.0. OLMo-1B averages 60.4. In Figure 1, accuracy rises with tokens except on OBQA, and the authors state annealing the learning rate to 0 at the end helps.
• Fit and adaptation: the authors state that given Paloma decontamination, OLMo-7B's bits per byte is competitive and exceeds other models on C4, while being less sample-efficient on sources farther from web crawls like Wikipedia. In Table 4, MMLU goes from 28.3 base to 47.3 with SFT (SFT+DPO is 46.2), ToxiGen toxic proportion from 81.4 to 1.7, AlpacaEval win rate with SFT+DPO is 69.3, and TruthfulQA is 52.0. In Table 8, GSM8k, BBH, TydiQA, and Codex-Eval are lower after DPO than SFT.
• Authors' conclusion: they call this the first OLMo release and state data and training have since improved, e.g., MMLU up 24 points to 52%. They intend to continue adding different scales, modalities, data, safety measures, and evaluations, while acknowledging English-only data, residual harmful content, incomplete failure logs, the Llama-oriented TÜLU mix, and that downstream tasks do not represent chat usage and automatic evaluation is noisy.