V.04 / DISCOVERY
Hypothesis testing
Which outcomes would lead an investigator to reject the null hypothesis at a specified error rate?
Start with the question.
Hypothesis testing organizes decisions under uncertainty by specifying alternatives, a test statistic, and error behavior. Neyman and Pearson's work supplies a historical foundation for this design perspective. A useful analysis starts before the data are inspected: which difference matters, what procedure will be applied, and how often it could produce a mistaken decision under the assumed conditions.
Distinguish the threshold from the evidence.
For an illustrative quality-control problem, define an acceptable defect rate and a sampling rule. Compare a small and a large sample with the same observed percentage of defects. Their uncertainty differs, so the same percentage can support different decisions. Record the practical cost of each kind of error before choosing a threshold.
What would you inspect?
- Null and alternative models, sample size, test statistic, and stopping rule.
- Type I error, power against a stated alternative, and the estimated effect size.
Take the next step.
Write a complete testing plan for a simple public dataset before calculating its result, then report the estimate and uncertainty alongside the decision.
Put part of the question to work.
These models isolate a mechanism you can vary. Their assumptions define the part of this topic they can illuminate.
Try evidence and base rates.
Change one parameter, compare the numerical output, and export the result. The experiment loads when you open it.
Watch the quantities change.
Evidence and base rates
Follow a screening result back through the population that produced it.
Read the explanation and method
A hypothetical batch contains 10,000 components. Detection is fixed at 99 percent and false flags at 5 percent.
The defect rate rises from 1 percent toward 20 percent. Orange shows actual defects that are flagged; cream shows sound items that are falsely flagged.
The posterior is true flags divided by all flags. Changing the starting population changes this fraction even when both screening rates stay fixed.
Method. Apply each conditional rate to its own population. True flags equal the defective population multiplied by the detection rate; false flags equal the sound population multiplied by the false-flag rate. Divide true flags by all flags to obtain the posterior probability.
Read the supporting material.
- On the problem of the most efficient tests of statistical hypotheses
Jerzy Neyman and Egon S. Pearson / Royal Society · The historical hypothesis-testing framework and error-rate design.
