Learning outcomes
By the end of this lesson you should be able to: state null and alternative hypotheses for a claim about a mean; compute a test statistic and a p-value; decide at a stated significance level and express the decision correctly; and explain what a p-value is not, including the Type I and Type II errors controlled by the significance level and the sample size.
Prerequisites
Lessons 7 and 8 (Sampling distributions & the CLT, Confidence intervals). You need the standard error and the normal distribution, and the two-sided critical values of lesson 8 are reused here as test statistics.
The big idea
A hypothesis test answers a narrow question with a precise tool. The p-value measures surprise under the null: if the observed sample mean would almost never arise when the null is true, the null loses credibility. What the p-value is not is a probability that the null is true — it is computed by assuming the null, so it cannot then tell you how likely the null is. Getting that one sentence right is most of statistical literacy.
The five steps, and the reasoning behind them
The procedure has five parts: state H0 and H1 together with the significance level alpha; compute the test statistic, a standardised distance from the null value; compute the p-value, the tail area beyond that distance; compare p with alpha and state the decision in context; and report the conclusion in the original units, never as a bare p-value.
The significance level alpha is the chance of rejecting H0 when H0 is actually true — a Type I error, or false alarm. Failing to reject H0 when it is false is a Type II error, a missed detection, and its probability falls as the sample size grows because the standard error shrinks and real effects become easier to see. The two error rates trade off at fixed n, which is why study design must consider both rather than only avoiding false alarms.
Watch out'Fail to reject H0' is not the same as 'H0 is true'. It means the data did not provide enough evidence against it, which is a statement about the evidence, not about the world. Absence of evidence is not evidence of absence.
Worked example 1: a complete z test for a mean
A factory claims its machines fill boxes to a mean of 50 g. A sample of 36 boxes averages 53 g with a standard deviation of 12. Test H0: mu = 50 against H1: mu is not 50 at the 5% level.
import math
n, xbar, s, mu0 = 36, 53, 12, 50
se = s / math.sqrt(n)
z = (xbar - mu0) / se
p_two = 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))
print("SE:", se)
print("z:", z)
print("two-sided p:", round(p_two, 4))
print("reject H0 at 5%:", p_two < 0.05)SE: 2.0
z: 1.5
two-sided p: 0.1336
reject H0 at 5%: FalseMethod: SE = 12 / 6 = 2, so z = (53 - 50) / 2 = 1.5. A deviation of 1.5 standard errors is unremarkable under the null — the two-sided p-value is 0.1336, well above 0.05 — so we fail to reject H0. The correct conclusion is that this sample does not provide convincing evidence that the mean fill differs from 50 g. It is not 'the mean is 50 g': the data are compatible with a range of true means, exactly the confidence interval of lesson 8.
Worked example 2: the same data with a larger sample
Keep the 2.5 g discrepancy but increase the sample from 36 to 100. Nothing about the effect changed; only the standard error did, and that flips the decision.
import math
mu0, xbar, s = 50, 52.5, 12
for n in [36, 100]:
se = s / math.sqrt(n)
z = (xbar - mu0) / se
p = 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))
print(f"n = {n}: SE = {se:.4f}, z = {z:.4f}, p = {p:.4f}, reject = {p < 0.05}")n = 36: SE = 2.0000, z = 1.2500, p = 0.2113, reject = False
n = 100: SE = 1.2000, z = 2.0833, p = 0.0372, reject = TrueMethod: with n = 36 the SE is 12 / 6 = 2, giving z = 2.5 / 2 = 1.25 and p = 0.2113; with n = 100 the SE is 12 / 10 = 1.2, giving z = 2.5 / 1.2 = 2.0833 and p = 0.0372, which is below 0.05. The warning this example carries: statistical significance is a function of sample size as much as of effect, so a smaller p-value does not automatically mean a bigger difference. A 2.5 g discrepancy that matters in practice matters equally at n = 36 — it is simply easier to detect at n = 100.
NoteStatistical significance and practical significance are different questions. The first asks whether the data is incompatible with H0; the second asks whether the size of the effect would change a decision. A half-gram difference can be 'significant' with enormous samples and still not matter to a single customer.
Worked example 3: one-sided versus two-sided
The choice of alternative changes the p-value by a factor of about two, and it must be made before the data is seen, not after.
| Hypotheses | Test statistic | P-value | Decision at 5% |
|---|---|---|---|
| H1: mu is not 50 (two-sided) | z = 1.5 | 0.1336 | Fail to reject |
| H1: mu > 50 (one-sided) | z = 1.5 | 0.0668 | Fail to reject |
| H1: mu > 50 with n = 100 | z = 2.0833 | 0.0186 | Reject |
Method: a two-sided p-value counts both tails, so it is twice the one-sided value for the same z: 2 x 0.0668 = 0.1336, and for z = 2.0833 the one-sided value is 0.0186 against a two-sided 0.0372. A one-sided test is only legitimate when the opposite direction is impossible or uninteresting in advance; choosing it after looking at the data halves the p-value without any new evidence, which is a form of p-hacking.
Common mistakes
Watch outReading p as the probability that H0 is true. The p-value is computed by assuming H0; it is the probability of data this extreme under the null, not the probability of the null itself.
Watch outWriting 'H0 is accepted' as the conclusion. The honest phrasing is 'fail to reject H0' — the test measures the evidence against the null, and a small sample may fail to reject a false null.
Watch outChoosing a one-sided test after seeing which way the data fell. The alternative must be fixed in advance; switching it afterwards turns a 5% test into a 10% test in disguise.
Watch outRunning many tests and reporting the smallest p-value. After 20 independent tests at the 5% level, one 'significant' result is expected by chance alone — which is why corrections for multiple testing exist.
Check your understanding
A test yields p = 0.03. What is the correct reading of this number?
Why does the same 2.5 g difference fail to reject at n = 36 but reject at n = 100?
What are the Type I and Type II errors in the box-filling test?
Multiple choice
A sample of 36 items has x-bar 53, s = 12, and H0 says mu = 50. What is the test statistic?
A test yields p = 0.03. Which statement is correct?
A test fails to reject H0 at the 5% level. Which conclusion is correct?
Why is picking a one-sided alternative after inspecting the data invalid?
Twenty independent tests are run at the 5% level. Roughly how many 'significant' results are expected by chance if all the nulls are true?
Summary and next
A test states H0, computes a standardised distance from it and reports the p-value — the probability, under H0, of data at least as extreme as observed. Compare p with the pre-set alpha to reject or fail to reject, then express the decision in context. A p-value is not the probability that H0 is true, 'fail to reject' is not 'accept', and significance depends on the sample size as much as on the effect. Type I error is controlled by alpha; Type II error by the sample size.
Next: inference for means. The machinery just built is applied in its proper form, with the t-distribution handling a standard deviation estimated from the sample, and with the conditions that make a one-sample interval or test valid checked before any number is reported.