Learning outcomes
By the end of this lesson you should be able to: fit a least-squares line to a small dataset and interpret its slope and intercept in context; compute residuals, the sum of squared errors and R squared for the fit; read a correlation between -1 and 1 correctly; and explain why a fitted line must not be used outside the range of the observed x values.
Prerequisites
Lessons 3 and 5 (Spread & shape, Random variables & expectation). Regression uses the mean and the standard deviation of two variables, and the idea of a best estimate from lesson 5 applied to a line instead of a single number.
The big idea
Regression replaces a cloud of points with one line and two numbers. Those two numbers — the slope and R squared — answer the two questions everyone actually asks: how much does y move when x moves, and how well does x explain y at all. Because the line is fitted to averages, it describes tendencies, and the residual for any single observation measures what the tendency misses.
Why the least-squares line is the one worth fitting
For any line a + bx, the residual of an observation is the observed y minus the fitted y-hat, and the sum of squared residuals is the total squared distance between the data and the line. Minimising that sum with respect to a and b gives a closed-form solution: the slope is the sum of (x - mean x)(y - mean y) divided by the sum of (x - mean x) squared, and the intercept is the mean of y minus the slope times the mean of x. Squaring is what makes the solution unique and easy — absolute values would also measure distance but have no derivative at zero.
Two structural facts about the residuals follow, and both are checkable on any fitted line: the residuals sum to zero, and they are uncorrelated with x. If either fails, the line is not the least-squares line. These are the same balancing properties the mean has, extended from one variable to a fitted relationship.
Worked example 1: fitting a line to five points
Five pairs of observations. The line is fitted by minimising the sum of squared residuals, and every quantity below is exact.
| x | y | y-hat (fitted) | residual (y - y-hat) |
|---|---|---|---|
| 1 | 3.0 | 2.8 | 0.2 |
| 2 | 3.0 | 3.4 | -0.4 |
| 3 | 4.0 | 4.0 | 0.0 |
| 4 | 5.0 | 4.6 | 0.4 |
| 5 | 5.0 | 5.2 | -0.2 |
import math
x = [1, 2, 3, 4, 5]
y = [3.0, 3.0, 4.0, 5.0, 5.0]
n = len(x)
mx, my = sum(x) / n, sum(y) / n
sxy = sum((xi - mx) * (yi - my) for xi, yi in zip(x, y))
sxx = sum((xi - mx) ** 2 for xi in x)
slope = sxy / sxx
intercept = my - slope * mx
print("slope:", slope, "intercept:", intercept)
print("fitted line: y-hat =", round(intercept, 2), "+", round(slope, 2), "x")
pred = [intercept + slope * xi for xi in x]
resid = [yi - pi for yi, pi in zip(y, pred)]
print("fitted values:", [round(p, 2) for p in pred])
print("residuals:", [round(r, 2) for r in resid])
print("sum of residuals:", round(sum(resid), 10) + 0.0)
print("sum of x times residual:", round(sum(xi * r for xi, r in zip(x, resid)), 10) + 0.0)
sse = sum(r * r for r in resid)
sst = sum((yi - my) ** 2 for yi in y)
print("SSE:", round(sse, 2), "SST:", round(sst, 2))
print("R^2:", round(1 - sse / sst, 4))
r = sxy / math.sqrt(sxx * sst)
print("correlation r:", round(r, 4))
print("prediction at x = 6:", round(intercept + slope * 6, 2))slope: 0.6 intercept: 2.2
fitted line: y-hat = 2.2 + 0.6 x
fitted values: [2.8, 3.4, 4.0, 4.6, 5.2]
residuals: [0.2, -0.4, 0.0, 0.4, -0.2]
sum of residuals: 0.0
sum of x times residual: 0.0
SSE: 0.4 SST: 4.0
R^2: 0.9
correlation r: 0.9487
prediction at x = 6: 5.8Method: the mean of x is 3 and the mean of y is 4.0. The sum of (x - 3)(y - 4.0) is 2 + 1 + 0 + 1 + 2 = 6, and the sum of (x - 3) squared is 4 + 1 + 0 + 1 + 4 = 10, so the slope is 6 / 10 = 0.6 and the intercept is 4.0 - 0.6 x 3 = 2.2. Reading the slope: for each one-unit increase in x, y rises by 0.6 units on average. The residuals sum to 0.0 and the sum of x times each residual is also 0.0, confirming the fit. SSE = 0.04 + 0.16 + 0 + 0.16 + 0.04 = 0.4 and SST = 1 + 1 + 0 + 1 + 1 = 4.0, so R squared = 1 - 0.4 / 4 = 0.9: the line accounts for 90% of the variation in y. The correlation is the positive square root of 0.9, about 0.9487, and a prediction at x = 6 is 2.2 + 0.6 x 6 = 5.8.
NoteThe correlation is positive because the slope is positive, and its size is the square root of R squared. A correlation of 0.95 does not mean 95% accuracy of prediction; it means R squared of 0.90, so 10% of the variation in y remains unexplained.
Worked example 2: residual structure reveals a wrong model
A straight line fitted to clearly curved data. The fit looks respectable by R squared and the residuals tell a different story — they are not scattered randomly.
x = [1, 2, 3, 4, 5]
y = [1.0, 4.0, 9.0, 16.0, 25.0]
n = len(x)
mx, my = sum(x) / n, sum(y) / n
sxy = sum((xi - mx) * (yi - my) for xi, yi in zip(x, y))
sxx = sum((xi - mx) ** 2 for xi in x)
slope = sxy / sxx
intercept = my - slope * mx
pred = [intercept + slope * xi for xi in x]
resid = [yi - pi for yi, pi in zip(y, pred)]
sse = sum(r * r for r in resid)
sst = sum((yi - my) ** 2 for yi in y)
print("slope:", slope, "intercept:", intercept)
print("fitted values:", [round(p, 2) for p in pred])
print("residuals:", [round(r, 2) for r in resid])
print("R^2:", round(1 - sse / sst, 4))slope: 6.0 intercept: -7.0
fitted values: [-1.0, 5.0, 11.0, 17.0, 23.0]
residuals: [2.0, -1.0, -2.0, -1.0, 2.0]
R^2: 0.9626Method: the fitted line is y-hat = -7.0 + 6.0x, with residuals 2.0, -1.0, -2.0, -1.0, 2.0 in the order of increasing x. A good fit leaves residuals with no pattern; here they are positive at both ends and negative throughout the middle, so a straight line is missing structure in the data. R squared of 0.9626 looks strong, and the residual pattern still forbids using the line — a reminder that a single summary number can conceal a badly wrong model.
Watch outNever use a fitted line to predict far outside the range of the observed x values. The five points above ran from x = 1 to x = 5; a prediction at x = 6 is a mild extrapolation to 5.8, but the fitted line also claims y = -1 at x = 1, which is impossible for a counting outcome. Beyond the observed range the relationship may not be linear at all.
Common mistakes
Watch outReading a correlation as causation. A strong slope between two variables says nothing about which causes which, or whether a third variable drives both; establishing causation needs a design, not a correlation.
Watch outTreating R squared as accuracy. R squared is the share of variation explained, not a percentage of correct predictions; a model with R squared 0.9 can still be badly wrong for individual cases, as the residuals show.
Watch outExtrapolating beyond the observed x range. The line is only supported where data exists; outside that range it asserts a relationship that was never observed.
Watch outMistaking a negative correlation for a weak one. Correlation lies between -1 and 1, and -0.95 indicates an association every bit as strong as +0.95, merely in the opposite direction.
Check your understanding
For the five-point fit above, what does the slope of 0.6 mean in context?
In the curved-data example, why does R squared of 0.9626 not justify using the straight line?
The correlation between x and y is -0.83. What is R squared, and how strong is the association?
Multiple choice
A least-squares line fitted to five (x, y) points gives y-hat = 2.2 + 0.6x. What does the slope 0.6 mean?
For the five-point fit, what are SSE, SST and R squared?
In the curved-data example, the residuals are +2, -1, -2, -1, +2. What do they indicate?
A correlation of -0.9 is reported. Which statement is correct?
Why must a fitted line not be used to predict far outside the observed range of x?
Summary and next
The least-squares line minimises the sum of squared residuals, with slope equal to the sum of (x - mean x)(y - mean y) over the sum of (x - mean x) squared and intercept equal to mean y minus slope times mean x. Its residuals sum to zero and are uncorrelated with x. R squared is one minus SSE over SST and equals the correlation squared, so it reports the share of variation the line explains. Patterns in residuals reveal a wrong functional form, a high R squared does not mean accurate individual predictions, and neither slope nor correlation establishes causation.
This closes the inference arc of the course: describe data, quantify uncertainty in that description, build intervals and tests on it, and finally model one variable as a function of another. Lessons 10 and 11 extend the same machinery to one and two samples, and the same discipline — state the assumptions, check them, then report in context — is what carries forward.