What Is Hypothesis Testing?
Hypothesis testing is a formal procedure for using sample data to decide between two competing claims about a population. It is the backbone of inferential statistics and appears in medicine (does this drug work?), manufacturing (is our process drifting?), economics, psychology, and any field where we need to make decisions under uncertainty.
The logic resembles a courtroom trial: the defendant is presumed innocent (the null hypothesis, H₀) unless the evidence proves otherwise. We then state an alternative hypothesis (H₁ or Hₐ), collect data, and compute how surprising the observed data would be if H₀ were true. If the data is sufficiently unlikely under H₀, we reject H₀; otherwise we fail to reject it.
Note the careful language: we never 'accept' the null hypothesis, because failing to find evidence against it is not the same as proving it true.
α — typically 0.05, 0.01, or 0.10 — which is the maximum risk of falsely rejecting a true null hypothesis we're willing to accept.Setting Up the Hypotheses
Every hypothesis test starts with a precisely worded null and alternative hypothesis. The null hypothesis (H₀) always contains an equality — =, ≤, or ≥ — and represents the status quo or 'no effect.' The alternative hypothesis (H₁) contains the strict inequality that corresponds to the claim you're trying to prove.
The three forms of the alternative
| Form | Hypotheses | When to use |
|---|---|---|
| Two-tailed test | H₀: μ = μ₀ vs H₁: μ ≠ μ₀ | Use when deviations in either direction matter. |
| Right-tailed test | H₀: μ ≤ μ₀ vs H₁: μ > μ₀ | Use when you're testing if the parameter is bigger than claimed. |
| Left-tailed test | H₀: μ ≥ μ₀ vs H₁: μ < μ₀ | Use when testing if it's smaller. |
A coffee shop claims its medium drink contains 12 oz. A customer suspects the shop is underfilling.
H₀: μ ≥ 12 vs H₁: μ < 12 (left-tailed, because underfilling means mean is less than 12)A factory's machine is calibrated to produce 500g boxes. Quality control tests whether the machine has drifted in either direction.
H₀: μ = 500 vs H₁: μ ≠ 500 (two-tailed)Key rule: the alternative hypothesis reflects the research question. Set it up BEFORE looking at data — choosing the tail based on data inflates error rates.
Test Statistics — Z-Test vs T-Test
A test statistic summarizes how far your sample result is from what H₀ predicts, measured in standard error units. Two common choices:
σ is known, or when the sample size is large (n ≥ 30) and you can use the sample standard deviation as an approximation. Formula: z = (x̄ − μ₀)/(σ/√n).σ is unknown and the sample size is small (n < 30), assuming the population is approximately normal. Formula: t = (x̄ − μ₀)/(s/√n), with n − 1 degrees of freedom.A light bulb manufacturer claims average lifetime 1200 hours, known σ = 100. A sample of 50 bulbs gives x̄ = 1180. Test H₀: μ = 1200 vs H₁: μ ≠ 1200 at α = 0.05.
z = (1180 − 1200)/(100/√50) = −20/14.14 ≈ −1.414.
Critical values for two-tailed test at α = 0.05: ±1.96.
Since −1.414 is within (−1.96, 1.96), fail to reject H₀.A researcher studies whether a new diet lowers cholesterol below the population mean of 210. Sample of 16 participants, x̄ = 198, s = 22. H₀: μ ≥ 210 vs H₁: μ < 210, α = 0.05.
t = (198 − 210)/(22/√16) = −12/5.5 = −2.18, df = 15.
One-tailed critical value: t_{0.05,15} = −1.753.
Since −2.18 < −1.753, reject H₀ — evidence supports that the diet lowers cholesterol.P-Values and Decision Rules
H₀ is true.p ≤ α, reject H₀. If p > α, fail to reject H₀.The p-value is one of the most misunderstood concepts in statistics, so be careful. It does NOT mean the probability that H₀ is true. It means: assuming H₀ is true, how likely would we be to see results this extreme by random chance? A small p-value means the observed result is surprising under H₀, so we reject H₀.
How to compute
| Test type | Formula |
|---|---|
| Two-tailed test | p = 2 · P(Z ≥ |z|) |
| One-tailed right test | p = P(Z ≥ z) |
| One-tailed left test | p = P(Z ≤ z) |
Use a standard normal (or t-distribution) table/calculator.
From the light bulb example, z = −1.414, two-tailed.
p = 2 · P(Z ≤ −1.414) = 2 · 0.0786 = 0.1572.
Since 0.1572 > 0.05, fail to reject H₀.From the cholesterol example, t = −2.18, df = 15, one-tailed left. From t-tables, p ≈ 0.023.
Since 0.023 < 0.05, reject H₀.Equivalence of methods: Either compare the test statistic to critical values, OR compare p-value to α. Both give the same conclusion. Reporting the p-value is preferred because it conveys the strength of evidence, not just the decision.
Type I and Type II Errors
Two types of errors can occur in hypothesis testing:
H₀ when it is actually true. Probability is α (the significance level). Also called a 'false positive.'H₀ when it is actually false. Probability is denoted β. Also called a 'false negative.' The complement 1 − β is the power of the test — the probability of correctly detecting a true effect.Trade-off: Decreasing α (being stricter about rejecting H₀) increases β and reduces power. The only way to decrease both error rates is to increase sample size.
In a medical trial for a new drug, H₀: 'drug has no effect.' A Type I error means concluding the drug works when it doesn't (putting ineffective treatments into circulation). A Type II error means missing a truly effective drug.
Which matters more depends on context: for life-threatening disease, missing an effective treatment may be worse than approving an ineffective one; for cosmetic products, the reverse.
Calculating β: Power calculations require specifying a specific alternative value μ₁. Example: test H₀: μ = 100 vs H₁: μ > 100 with σ = 15, n = 36, α = 0.05.
Critical value: x̄ > 100 + 1.645 · (15/6) = 104.11.
If true mean is μ₁ = 105, power = P(X̄ > 104.11 | μ = 105) = P(Z > (104.11 − 105)/2.5) = P(Z > −0.356) ≈ 0.639.
β = 1 − 0.639 = 0.361Larger sample sizes, larger effect sizes, and larger α all increase power.
Complete Worked Example and Common Pitfalls
A cereal company's packaging machine is supposed to fill boxes to 450g on average. Quality control randomly samples 25 boxes and finds x̄ = 445g, s = 12g. Test at α = 0.01 whether the machine needs calibration.
- Step 1 — hypotheses
H₀: μ = 450vsH₁: μ ≠ 450(two-tailed). - Step 2 — test choice
Small sample (25), σ unknown → t-test, df = 24.
- Step 3 — test statistic
t = (445 − 450)/(12/√25) = −5/2.4 ≈ −2.08. - Step 4 — p-value
Two-tailed p for
|t| = 2.08, df = 24: p ≈ 0.048. - Step 5 — decision
0.048 > 0.01, fail to reject
H₀. At the 1% level, there's not enough evidence to conclude the machine is miscalibrated (though at α = 0.05 we would reject). This demonstrates how the choice of α matters: stricter thresholds require stronger evidence.
Common pitfalls
(1) Choosing α after seeing the data — this inflates Type I error.
(2) Confusing statistical significance with practical significance: with a huge sample, tiny differences become 'significant' but may be meaningless.
(3) Running many tests without correction (the 'multiple comparisons problem'): with 20 tests at α = 0.05, you'd expect 1 false positive just by chance.
(4) Believing a high p-value proves H₀ — it doesn't; it just means we lack evidence to reject it.
(5) Interpreting p as the probability that H₀ is true — incorrect.
If you're working through hypothesis tests for a class or research project, scan them with Solver AI — it sets up the correct hypotheses, picks z vs t, computes the statistic, the p-value, and shows the full decision logic.