Learning outcomes
By the end of this lesson you should be able to: compute the range, the quartiles, the interquartile range, the variance and the standard deviation of a small numerical dataset; explain why the sample variance divides by n - 1 rather than n; choose a measure of spread that survives an outlier; and read skew off the relationship between the mean and the median.
Prerequisites
Lesson 2 (Summarizing the center). You need the mean and the median, because every measure of spread below is defined as a distance from one of them.
The big idea
Two datasets can share a mean of 50 and be nothing alike: one where nearly every value lies between 48 and 52, and another where values are scattered from 0 to 100. The center says where the data sits; the spread says how far it stretches. Reporting one without the other is how a summary gets misread, which is why a serious report always quotes both.
Why the sample variance divides by n - 1
Start from the definition. The variance is the average squared distance from the mean: subtract the mean from each value, square the result, and average those squares. Squaring serves two purposes — it makes every contribution positive, so values below and above the mean cannot cancel, and it makes large deviations weigh more than small ones, which is the behaviour we want from a measure of spread.
There is one subtlety. When the data is a sample used to estimate the spread of a wider population, you divide the sum of squares by n - 1, not n. The reason is that the sample mean is computed from the same data, so it sits slightly closer to the observations than the population mean would; the deviations from it are therefore slightly too small on average. Dividing by n - 1 removes that tilt, giving an unbiased estimate of the population variance. The n - 1 is called the degrees of freedom: n observations, one constraint (the mean must be matched), leaving n - 1 free pieces of information about spread.
The correction is small when n is large — with 500 observations, dividing by 499 or by 500 barely differs — and it matters a great deal when n is small. Use n - 1 whenever the data is a sample; use n only when you hold the whole population and are describing it rather than inferring from it.
Worked example 1: variance and standard deviation from a tiny dataset
The dataset from lesson 2: 4, 5, 5, 6, 6, 6, 7, 7, 8, with a mean of 6. Every squared deviation is small, so the arithmetic can be followed line by line.
import math
data = [4, 5, 5, 6, 6, 6, 7, 7, 8]
n = len(data)
mean = sum(data) / n
sq = [(x - mean) ** 2 for x in data]
var_sample = sum(sq) / (n - 1)
var_pop = sum(sq) / n
print("n:", n)
print("mean:", mean)
print("squared deviations:", sq)
print("sum of squared deviations:", sum(sq))
print("sample variance (sum / (n - 1)):", var_sample)
print("population variance (sum / n):", round(var_pop, 4))
print("sample SD (sqrt of the sample variance):", round(math.sqrt(var_sample), 4))
print("range:", max(data) - min(data))n: 9
mean: 6.0
squared deviations: [4.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 4.0]
sum of squared deviations: 12.0
sample variance (sum / (n - 1)): 1.5
population variance (sum / n): 1.3333
sample SD (sqrt of the sample variance): 1.2247
range: 4Method: the deviations from the mean of 6 are -2, -1, -1, 0, 0, 0, 1, 1, 2, whose squares are 4, 1, 1, 0, 0, 0, 1, 1, 4. They sum to 12. Dividing by n - 1 = 8 gives a sample variance of exactly 1.5, and the square root of 1.5 is about 1.2247, so a typical value sits roughly 1.2 units from the mean. The population variance would be 12 / 9 = 1.3333, slightly smaller because the correction is missing. The range is 8 - 4 = 4, which coincidentally looks similar to the SD here because the data is symmetric and compact.
Worked example 2: quartiles, the IQR and the outlier fence
The standard deviation uses every value, so one extreme observation can inflate it. The quartiles give a robust alternative: split the sorted data into quarters and measure the spread of the middle half.
| Summary statistic | Value | How it is obtained |
|---|---|---|
| Minimum | 4 | The smallest value |
| Q1 (first quartile) | 5 | Median of the lower half (4, 5, 5, 6) → 5 |
| Median | 6 | The middle value of all nine |
| Q3 (third quartile) | 7 | Median of the upper half (6, 7, 7, 8) → 7 |
| Maximum | 8 | The largest value |
| IQR | 2 | Q3 - Q1 = 7 - 5 |
| Range | 4 | Maximum - minimum = 8 - 4 |
data = sorted([4, 5, 5, 6, 6, 6, 7, 7, 8])
n = len(data)
median = data[n // 2]
lower = data[: n // 2]
upper = data[n // 2 + 1 :]
q1 = lower[len(lower) // 2]
q3 = upper[len(upper) // 2]
iqr = q3 - q1
print("sorted:", data)
print("Q1:", q1, "median:", median, "Q3:", q3)
print("IQR:", iqr)
print("fences:", q1 - 1.5 * iqr, "to", q3 + 1.5 * iqr)
print("values outside the fences:", [x for x in data if x < q1 - 1.5 * iqr or x > q3 + 1.5 * iqr])sorted: [4, 5, 5, 6, 6, 6, 7, 7, 8]
Q1: 5 median: 6 Q3: 7
IQR: 2
fences: 2.0 to 10.0
values outside the fences: []Method: sort the nine values and take the fifth (6) as the median. The four values below it are 4, 5, 5, 6, whose middle two are 5 and 5, so Q1 = 5. The four values above it are 6, 7, 7, 8, whose middle two are 7 and 7, so Q3 = 7. The IQR is therefore 7 - 5 = 2, and the outlier fences sit at Q1 - 1.5 x 2 = 2 and Q3 + 1.5 x 2 = 10. Nothing falls outside them, so this dataset has no outliers by the standard rule.
The fence formula is a convention that works because the middle 50% of roughly bell-shaped data lives within about 1.5 IQRs of the quartiles, so values further out are rare enough to be worth flagging. It marks a value for inspection; it is not a verdict. In a real dataset an extreme value is often the most informative case in the file.
Worked example 3: what one big value does to spread and shape
Add a single value of 50 to the dataset and recompute. This is the same dataset lesson 2 used to show the mean moving away from the median.
import math
data = sorted([4, 5, 5, 6, 6, 6, 7, 7, 8, 50])
n = len(data)
mean = sum(data) / n
median = (data[n // 2 - 1] + data[n // 2]) / 2
q1 = data[n // 4]
q3 = data[(3 * n) // 4]
sq = [(x - mean) ** 2 for x in data]
print("mean:", mean)
print("median:", median)
print("Q1:", q1, "Q3:", q3, "IQR:", q3 - q1)
print("mean minus median:", round(mean - median, 1))
print("sample SD:", round(math.sqrt(sum(sq) / (n - 1)), 2))
print("sum of squared deviations:", round(sum(sq), 2))mean: 10.4
median: 6.0
Q1: 5 Q3: 7 IQR: 2
mean minus median: 4.4
sample SD: 13.96
sum of squared deviations: 1754.4Method: the ten values sum to 104, so the mean is 104 / 10 = 10.4. Sorted, the two middle values are both 6, so the median is 6.0, and the quarter values are unchanged at 5 and 7, so the IQR is still 2. The squared deviations from 10.4 now include (50 - 10.4) squared = 1568.16, and the total sum of squares leaps to 1754.4, pushing the sample SD to 13.96 — more than eleven times its original value of 1.2247.
This is the diagnostic pair worth memorising: a mean far above the median, together with a standard deviation larger than half the mean, is the signature of right skew. Income, house prices, hospital stay lengths and reaction times all behave this way, which is why they are reported with medians and IQRs rather than means and standard deviations.
Choosing a measure of spread
| Measure | Uses | Affected by an outlier? | Best used when |
|---|---|---|---|
| Range | Only the two extremes | Yes, completely | A quick sense of the total span |
| IQR | The middle 50% | No | The data is skewed or contains outliers |
| Standard deviation | Every value | Yes, but less than the range | The data is roughly symmetric |
| Variance | Every value, squared | Yes, strongly | As a stepping stone to the SD, not as a report |
NoteReport the variance only when you need a quantity that adds up — for example, the variance of a sum of independent variables is the sum of their variances. To describe data, take its square root and report the standard deviation, which is back in the data's own units.
Common mistakes
Watch outDividing by n when estimating spread from a sample. The sample variance divides the sum of squares by n - 1; using n gives a systematically small estimate, and the damage is worst exactly when you have least data.
Watch outQuoting a standard deviation without units. An SD of 13.96 means nothing on its own; an SD of 13.96 minutes is a statement about spread.
Watch outUsing the standard deviation on heavily skewed data. The SD is pulled by the long tail just as the mean is, so for skewed data report the median together with the IQR instead.
Watch outConfusing the IQR with the range. The range spans the whole dataset; the IQR spans only the middle half. A dataset with a huge range and a small IQR is precisely the outlier case.
Check your understanding
The data {2, 4, 4, 4, 5, 5, 7, 9} has mean 5. What is its sample variance, and what is its standard deviation?
Why does the sample variance divide by n - 1 rather than n?
A hospital records length of stay, which is strongly right-skewed. Which pair of statistics should it report?
Multiple choice
What is the sample variance of {2, 4, 4, 4, 5, 5, 7, 9}?
Every value in a dataset is multiplied by 10. What happens to the standard deviation and the variance?
Which measure of spread is unaffected by a single extreme outlier?
A distribution of incomes has a mean well above its median. What does this indicate?
Why is the variance rarely reported as the headline summary of spread?
Summary and next
Spread answers the question the center leaves open. The range is max minus min and the IQR is Q3 minus Q1; both are robust, and the IQR is the more informative of the two. The variance averages the squared deviations from the mean — dividing by n - 1 for a sample — and the standard deviation is its square root, back in the original units. For skewed data pair the median with the IQR; for symmetric data, the mean with the SD.
Next: probability rules. Describing a dataset you already hold is the first half of statistics. From here on you will reason about data you have not yet seen, which requires the rules for combining chances — unions, complements, intersections and conditional probability — and that is the whole of lesson 4.