STRATA
Skip to study
XLVII

IV.11 / AI

Scaling laws, Kaplan, and Chinchilla

At a fixed training budget, how does allocating computation between examples and parameters affect loss?

A source-led introduction2 source references0 related experiments

Start with the question.

Scaling-law studies fit relationships between model performance and resources over measured ranges. Kaplan and colleagues and the later Chinchilla study provide distinct experiments and compute-allocation conclusions. Their comparison is useful because the resource budget is not a single number: parameters, training tokens, optimization, and evaluation distributions all matter. An empirical curve should retain the conditions under which it was fitted.

Compare two uses of a fixed budget.

Imagine a fixed training budget divided between a larger model with fewer token exposures and a smaller model trained on more data. Plot a hypothetical loss for each allocation and label the measured region separately from any extrapolation. The decision depends on the fitted relationship and the intended evaluation, not parameter count alone.

What would you inspect?

  • Training compute, parameter count, token count, and validation-loss definition.
  • The fitted range, residuals, and sensitivity of the recommended allocation.

Take the next step.

Recreate a small comparison table of the two papers' experimental assumptions and identify which changed assumption matters most for their allocation result.

Read the supporting material.

  1. Scaling Laws for Neural Language Models

    Jared Kaplan and coauthors · Empirical language-model loss scaling under specified experiments.

  2. Training Compute-Optimal Large Language Models

    Jordan Hoffmann and coauthors · Training-compute allocation across parameter and token counts.

Continue through the versioned study records.

Follow the comparisons.

Read the neighboring questions.