STRATA
Skip to study
XLVII

SOURCE RECORD / 2022

Compute has a budget

DocumentedR01 · 2026-11-18
Scope of this record

KNOWN refers to the reported empirical training-budget study. It is not a universal optimum for every dataset, architecture or deployment objective.

Statement

Hoffmann and colleagues investigate how a fixed training-compute budget should be divided between model size and training tokens, then test their allocation with Chinchilla. [source]

Equation

Unrecorded

The connection is scoped to this source. No broader cross-domain equality is established by this entry.

Observation

The abstract describes runs across model sizes and token budgets. Its central result concerns the allocation of training resources rather than minimizing all downstream serving costs. [source]

Progressive questions

What is fixed: training operations, wall time, memory or money? Which token distribution is assumed? How do inference volume and deployment constraints change the preferred model even when training compute is held constant?

Source response

The authors report that their tested allocation favors more training data than several earlier large-model configurations used. The scope is the studied training regime. [source]

Interpretation

XLVII reading: a tradeoff viewer should distinguish an observed run from a fitted trend and a user-selected extrapolation. It should never convert a scaling curve into a promise about revenue or general capability.

Conclusion

The paper gives a resource-allocation result to examine, not a timeless numerical rule for all learning systems. [source]

Next experiment

Proposed: use a small published benchmark grid with complete compute and data accounting. Compare an interpolated recommendation with a held-out run. Show uncertainty and out-of-range requests explicitly; do not train a large model merely to animate the curve.

Work through a related example.

IV.11 / Scaling laws, Kaplan, and Chinchilla

Compare two uses of a fixed budget.

Imagine a fixed training budget divided between a larger model with fewer token exposures and a smaller model trained on more data. Plot a hypothetical loss for each allocation and label the measured region separately from any extrapolation. The decision depends on the fitted relationship and the intended evaluation, not parameter count alone.

IV.16 / Efficient learning and inference

Estimate storage before measuring speed.

A billion weights stored at 16 bits require about two billion bytes for the weights alone. An idealized four-bit representation reduces that component to about half a billion bytes, before scales, metadata, and runtime state. Compare this estimate with measured process memory and latency; a storage reduction need not translate proportionally into faster execution.

X.06 / Optimization, control, and decision theory

Solve a small allocation problem.

Let x and y be nonnegative activity levels with x + y ≤ 4, x ≤ 2, and y ≤ 3. Maximize the illustrative value 3x + 2y. The feasible choice x = 2, y = 2 gives value 10; compare the other corner points to verify the optimum. Then change one coefficient and inspect whether the preferred allocation changes.

Follow earlier and later work.

Before

  • Proposed connection · UNKNOWNAttention without recurrence

    Editorial research connection to investigate; not a verified correspondence, causal lineage, or mathematical equivalence.

  • Proposed connection · UNKNOWNWhat survives a replication?

    Editorial research connection to investigate; not a verified correspondence, causal lineage, or mathematical equivalence.

After

No linked record.

Follow the chain
  1. Attention without recurrence · 1 link away · UNKNOWN
  2. What survives a replication? · 1 link away · UNKNOWN
  3. Prediction and residual error · 2 links away · UNKNOWN
  4. Compression, reopened · 2 links away · UNKNOWN
  5. Memory as a collective state · 3 links away · UNKNOWN
  6. The chemical compass · 3 links away · UNKNOWN
  7. Compression → classification · 3 links away · UNKNOWN
  8. Information is not meaning · 4 links away · UNKNOWN
  9. Dynamic Markov modelling · 4 links away · KNOWN
  10. Erasure has conditions · 5 links away · UNKNOWN
  11. Reversibility and the record · 6 links away · UNKNOWN

Placement.

Technology

This entry records the cited source and a proposed next investigation. It makes no claim of historical abandonment, suppression, or independent reproduction.

Sources.

  1. Training Compute-Optimal Large Language Models

    Jordan Hoffmann et al. / author manuscript · arXiv:2203.15556; original 29 March 2022; abstract.

    Abstract and metadata inspected; the scaling-fit data and all training runs were not reanalyzed. Checked 2026-11-18.

Record history
  1. 2026-11-18 / R1 / SOURCE_SEEDED

    Source-limited editorial seed with explicit inspection scope. Research questions and proposed experiments are not executed results.