Questions on least squares regression and residuals

Question 1

Define a residual, and state the quantity the least squares line makes as small as possible.

Accepted Answer

A residual is the observed y minus the y the line predicts at the same x: a vertical gap, not a perpendicular distance. The least squares line minimises the sum of the squares of those residuals over all the data points. B1 for the residual as a vertical gap, B1 for the sum of squared residuals.

Question 2

For the points (2, 5), (4, 9), (6, 10), (8, 14), (10, 17), find the regression line of y on x.

Accepted Answer

Σx = 30, Σy = 55, Σx² = 220 and Σxy = 388, so S_xx = 220 − 900/5 = 40 and S_xy = 388 − 330 = 58. Then b = 58/40 = 1.45 and a = 11 − 1.45 × 6 = 2.3, giving y = 2.3 + 1.45x. M1 for S_xx and S_xy, A1 for the gradient, A1 for the intercept.

Question 3

State which of the two variables is treated as the explanatory variable in a regression of y on x, and what that decision means for the gaps the method makes as small as possible.

Accepted Answer

x is the explanatory variable, the one set or controlled, and y is the response, the one measured. B1. The method therefore measures every gap vertically, in the y direction, at the observed value of x: it is the sum of the squares of those vertical gaps that is minimised, not the perpendicular distances to the line. B1.
Regressing x on y instead would minimise horizontal gaps and produce a different line from the same data, so the choice has to follow the way the data were collected.

Question 4

A set of readings gives S_xx = 25, S_xy = 60, x̄ = 8 and ȳ = 30. Find the equation of the regression line of y on x.

Accepted Answer

The gradient is b = S_xy/S_xx = 60/25 = 2.4. M1 for the formula, A1 for 2.4.
The line passes through (x̄, ȳ), so a = ȳ − bx̄ = 30 − 2.4 × 8 = 30 − 19.2 = 10.8, giving y = 10.8 + 2.4x. A1 for the intercept and the line.
Check by substituting x = 8: the line gives 10.8 + 19.2 = 30, which is ȳ, as every least squares line of y on x must.

Question 5

The regression line of y on x for the points (2, 5), (4, 9), (6, 10), (8, 14), (10, 17) is y = 2.3 + 1.45x. Find the five residuals and the residual sum of squares, checking the total two ways.

Accepted Answer

Fitted values are 5.2, 8.1, 11.0, 13.9 and 16.8, so the residuals are −0.2, 0.9, −1.0, 0.1 and 0.2. They sum to zero, as they must, which is a free check on the arithmetic. Squaring and adding: 0.04 + 0.81 + 1.00 + 0.01 + 0.04 = 1.9.
By formula, S_yy = 691 − 605 = 86, so the residual sum of squares is S_yy − S_xy²/S_xx = 86 − 58²/40 = 86 − 84.1 = 1.9. M1 for the fitted values, A1 for the five residuals, M1 for squaring and adding, A1 for the residual sum of squares.

Question 6

A test score y is recorded against x hours of practice for five students, giving (2, 5), (4, 9), (6, 10), (8, 14) and (10, 17). The regression line is y = 2.3 + 1.45x. Interpret both coefficients, predict the score for 7 hours, and say why a prediction for 20 hours would be unwise.

Accepted Answer

The gradient says that each extra hour of practice is associated with about 1.45 more marks. The intercept, 2.3 marks with no practice at all, comes from x = 0, which lies outside the observed range of 2 to 10 hours, so it should be quoted with care.
At x = 7 the prediction is 2.3 + 10.15 = 12.45 marks.
At x = 20 the model is being extrapolated to double the largest observation. There is no evidence that the relationship stays linear that far out, and in this context it cannot, since scores are bounded. B1 for interpreting the gradient, B1 for interpreting the intercept, B1 for the prediction, B1 for the extrapolation comment.

Question 7

Describe what a plot of residuals against x should look like if a straight-line model is appropriate, and what two patterns would argue against it.

Accepted Answer

It should show random scatter about zero with no trend and roughly constant spread. Systematic curvature, so that residuals are negative at the ends and positive in the middle or the reverse, means the relationship is not linear. A spread that widens with x means the assumption of constant variability fails. A single very large residual instead flags a possible outlier. B1 for random scatter about zero, B1 for curvature ruling out a linear model, B1 for a widening spread.

Question 8

Five points are recorded as (2, 5), (4, 9), (6, 10), (8, 14) and (10, 17), and the regression line of y on x is y = 2.3 + 1.45x. The last point is then found to have been misrecorded, and should read (10, 30). Recalculate the regression line and comment on the effect.

Accepted Answer

Σy becomes 68 and Σxy becomes 518, so S_xy = 518 − 30 × 68/5 = 518 − 408 = 110, while S_xx is unchanged at 40 because the x values have not moved.
Then b = 110/40 = 2.75 and a = 13.6 − 2.75 × 6 = −2.9, giving y = −2.9 + 2.75x.
One faulty reading has nearly doubled the gradient and pushed the intercept below zero. The least-squares method squares the residuals, so a single large miss counts for far more than several small ones. M1 for recomputing the sums, A1 for the new S_xy, A1 for the new line, B1 for the comment on the effect.

Question 9

The regression line of y on x for a set of six points is y = 4.5 + 1.8x. Three of the points are (3, 10.2), (5, 12.4) and (7, 21.3). Find the residual at each of the three, and state which of them the line describes least well.

Accepted Answer

The fitted value at each x is 4.5 + 1.8x, giving 9.9, 13.5 and 17.1. M1 for the fitted values.
The residual is observed minus fitted:
at (3, 10.2), 10.2 − 9.9 = 0.3;
at (5, 12.4), 12.4 − 13.5 = −1.1;
at (7, 21.3), 21.3 − 17.1 = 4.2. A1 for the first two, A1 for the third.
The line describes (7, 21.3) least well: its residual is nearly four times the size of the next largest, and the point sits well above the line. B1 for naming it with the reason.
A residual that stands out like that marks the point as a possible outlier, worth checking against the record before the fit is used. Subtract in the order observed minus fitted, since the sign carries the information about which side of the line the point lies.

Question 10

For a set of eight points, S_xx = 50, S_xy = 90 and S_yy = 200. Find the residual sum of squares, and state what a residual sum of squares of zero would mean.

Accepted Answer

RSS = S_yy − S_xy²/S_xx. M1 for the formula.
S_xy²/S_xx = 8100/50 = 162, the part of the total variation in y that the line accounts for. A1.
So RSS = 200 − 162 = 38. A1.
A residual sum of squares of zero would mean every residual is zero, so every point lies exactly on the line and the fit is perfect, which is the same as r = ±1. B1.
Here r = 90/√(50 × 200) = 0.9, and the 38 that is left over is what the straight line does not explain. Since S_xy²/S_xx cannot be negative, a residual sum of squares above S_yy is a sign to go back over the arithmetic.

Question 11

Annual maintenance cost y pounds is regressed on age x years for twelve machines aged between 1 and 9 years, giving y = 120 + 46x with a residual sum of squares of 21 400. One machine aged 6 years cost 512 pounds to maintain. Find the residual for that machine, interpret its sign, and state whether the machine is unusual, with a reason.

Accepted Answer

The fitted cost at x = 6 is 120 + 46 × 6 = 396 pounds. M1.
The residual is 512 − 396 = 116 pounds. A1.
The sign is positive, so this machine cost more to maintain than the line predicts for a machine of its age. B1 for the sign read back into the context.
For a sense of scale, the twelve residuals have squares totalling 21 400. A fitted line costs two degrees of freedom, one for the gradient and one for the intercept, so the residual variance is 21 400/10 = 2140 and a typical residual is about √2140 = 46 pounds. A residual of 116 is two and a half times that, so this machine is unusual and worth investigating. B1 for the comparison with a typical residual.

Question 12

Five pairs of readings are coded using p = (x − 6)/2 and q = y − 10, giving the coded points (−2, −5), (−1, −1), (0, 0), (1, 4) and (2, 7). Find the regression line of q on p, and hence find the regression line of y on x.

Accepted Answer

Σp = 0, Σq = 5, Σp² = 10 and Σpq = 29. Since Σp = 0 the correction terms vanish: S_pp = 10 and S_pq = 29.
So the coded gradient is 29/10 = 2.9, and with p̄ = 0 and q̄ = 1 the coded intercept is 1. The coded line is q = 1 + 2.9p.
Now decode. Substitute q = y − 10 and p = (x − 6)/2 into that line: y − 10 = 1 + 2.9(x − 6)/2 = 1 + 1.45(x − 6) = 1.45x − 7.7.
So y = 2.3 + 1.45x.
M1 for the coded sums, A1 for S_pp and S_pq, M1 for the coded gradient and intercept, A1 for the coded line, M1 for decoding, A1 for the line of y on x.
Dividing x by 2 multiplied the gradient by 2, so decoding halves it back to 1.45. Subtracting 10 from y and 6 from x moved the intercept, and only substituting back recovers it. Coding is chosen to make Σp = 0 and the numbers small, and it never changes the fitted line, only the arithmetic needed to reach it.

Question 13

Ten pairs of readings give Σx = 60, Σy = 250, Σx² = 424, Σy² = 6710 and Σxy = 1652. Find the regression line of y on x, and find the residual sum of squares.

Accepted Answer

Build the three summary statistics first, each correction term dividing by n = 10.
S_xx = 424 − 60²/10 = 424 − 360 = 64.
S_xy = 1652 − (60)(250)/10 = 1652 − 1500 = 152.
S_yy = 6710 − 250²/10 = 6710 − 6250 = 460.
M1 for the three formulas, A1 for all three values.
The gradient is b = 152/64 = 2.375, and with x̄ = 6 and ȳ = 25 the intercept is a = 25 − 2.375 × 6 = 25 − 14.25 = 10.75. M1 for b and a, A1 for y = 10.75 + 2.375x.
Then RSS = S_yy − S_xy²/S_xx = 460 − 152²/64 = 460 − 361 = 99. M1 for the formula, A1 for 99.
Two checks: 99 is positive and smaller than S_yy = 460, as it must be, and the line passes through (6, 25), since 10.75 + 14.25 = 25.

Question 14

Five readings are (1, 4), (2, 7), (3, 9), (4, 12) and (5, 28). Find the regression line of y on x and the five residuals. The reading at x = 5 is then suspected of being a recording error. Refit the line to the other four readings, find the residual the new line gives the suspect reading, and comment on what the two calculations together show.

Accepted Answer

All five points. Σx = 15, Σy = 60, Σx² = 55 and Σxy = 233, so S_xx = 55 − 225/5 = 10 and S_xy = 233 − (15)(60)/5 = 53. M1. b = 53/10 = 5.3 and a = 12 − 5.3 × 3 = −3.9, so the line is y = −3.9 + 5.3x. A1. The fitted values are 1.4, 6.7, 12.0, 17.3 and 22.6, so the residuals are 2.6, 0.3, −3.0, −5.3 and 5.4, which sum to zero. A1. Without the fifth point. Σx = 10, Σy = 32, Σx² = 30 and Σxy = 93, so S_xx = 5 and S_xy = 13. M1. b = 13/5 = 2.6 and a = 8 − 2.6 × 2.5 = 1.5, so the reduced line is y = 1.5 + 2.6x. A1. At x = 5 it predicts 1.5 + 13 = 14.5, so the suspect reading has residual 28 − 14.5 = 13.5. The comment. Against the full line the suspect point's residual was 5.4, only a little larger than the −5.3 at x = 4, so it did not stand out. Against the line built without it the same point is 13.5 away. B1 for the comparison. A point far from the pattern pulls the line towards itself, which shrinks its own residual and inflates everyone else's, so a residual plot from the full fit can understate how far out an influential reading is.

Question 15

The five residuals from a least squares regression of y on x are reported as 0.4, −1.2, 0.9, −0.5 and 0.6. Show that at least one of the five must be wrong, state the property of the least squares line that makes this certain, and, given that only the fourth has been misreported, find its correct value.

Accepted Answer

Add them: 0.4 − 1.2 + 0.9 − 0.5 + 0.6 = 0.2. M1 for adding the residuals, A1 for the total 0.2.
The residuals from a least squares line of y on x always sum to zero, so a set totalling 0.2 cannot be the residuals of any such fit and at least one figure is wrong. B1.
The reason is the intercept. Since a = ȳ − bx̄, the line passes through (x̄, ȳ), and summing the residuals gives Σy − na − bΣx = nȳ − n(ȳ − bx̄) − nbx̄ = 0. B1 for the line through the mean point as the reason.
If only the fourth is wrong, it must absorb the whole discrepancy: it should be 0.2 smaller, that is −0.5 − 0.2 = −0.7. A1.
It says nothing about whether the model is a good one: residuals summing to zero is a property of the fitting method, not evidence that a straight line belongs here.