STRATA
Skip to study
XLVII

V.05 / DISCOVERY

Information gain and experimental design

Which affordable measurement would most reduce uncertainty about the decision at hand?

A source-led introduction1 source references1 related experiments

Start with the question.

Experimental design asks which observation is worth acquiring next. Lindley's information-based treatment supplies a historical anchor for comparing experiments by what they are expected to reveal. The practical decision also includes cost, feasibility, and the action the result could change. An experiment is valuable when its possible outcomes separate relevant explanations rather than merely producing another measurement.

Compare two questions with four possible answers.

Assume four hypotheses are initially equally plausible. A noiseless question that splits them into two pairs yields one bit of information. A question that isolates one hypothesis from the other three yields about 0.811 bits on average. Then add unequal costs or noisy answers and inspect whether the preferred question changes.

What would you inspect?

  • Prior uncertainty, possible outcomes, their probabilities, and expected information gain.
  • Experiment cost and whether its outcome changes the intended decision.

Take the next step.

Design two candidate measurements for one unresolved atlas question and compare their predicted outcomes before choosing which measurement to prioritize.

Put part of the question to work.

These models isolate a mechanism you can vary. Their assumptions define the part of this topic they can illuminate.

Try evidence and base rates.

Change one parameter, compare the numerical output, and export the result. The experiment loads when you open it.

Open its full page

Watch the quantities change.

12 seconds · Original model visualization · Silent

Evidence and base rates

Follow a screening result back through the population that produced it.

Read the explanation and method

A hypothetical batch contains 10,000 components. Detection is fixed at 99 percent and false flags at 5 percent.

The defect rate rises from 1 percent toward 20 percent. Orange shows actual defects that are flagged; cream shows sound items that are falsely flagged.

The posterior is true flags divided by all flags. Changing the starting population changes this fraction even when both screening rates stay fixed.

Method. Apply each conditional rate to its own population. True flags equal the defective population multiplied by the detection rate; false flags equal the sound population multiplied by the false-flag rate. Divide true flags by all flags to obtain the posterior probability.

Read the supporting material.

  1. On a Measure of the Information Provided by an Experiment

    Dennis V. Lindley / Annals of Mathematical Statistics · An information-based approach to comparing experiments; the illustrative exercise here is independently constructed.

Follow the comparisons.

Read the neighboring questions.