# Why does regression dilution always bias the slope toward zero?

Regression dilution, in the case of linear regression, is supposed to be what happens when there is noise in the independent variables, namely the slope of the fitted linear regression model becomes smaller.

I don't understand why it can't also become larger. See the following toy example with only two data points, one of which is noisy.

The two black dots represent data points devoid of noise. Since there are only two data points, a linear regression model will fit them perfectly. If the noise changes the second point turning it into the yellow point, then indeed the slope diminishes. If, on the other hand, the noise changes the second point into the red point, the slope increases.

What am I getting wrong here?

• "Noise in the independent variables" refers to independent random variation of the same magnitude in all instances of an independent variable. It does not cover your case of noise in some instances but not in others. That requires a substantially more complicated variation of the regression model.
– whuber
Commented Jan 25, 2022 at 16:10
• @whuber I see, but what if the first observation is then shifted to the right and the second one to the left? That would increase the slope even more? Commented Jan 25, 2022 at 16:37
• I don't think you have the right picture of regression dilution. you should think of it more as a blur of values (and affecting both your points) - it is not saying that for any distortion of your inputs your slope will drop. It's more you measure 2 points 100 times and calculate the slope. Commented Jan 25, 2022 at 16:47
• @seanv507 Okay, I understand, you measure 2 points 100 times, calculate the slope, and I assume if you average them then regression dilution means that the slope will be smaller. Still, why would it be smaller and not the same? Say any point can be shifted with the same probability to the left or the right. Then I expect in half the cases the slope to be larger, in half the cases the slope to be smaller, right? Commented Jan 25, 2022 at 16:53

It's not obvious the usual argument won't go through with two points and no error in $$Y$$, and the result would be interesting either way. So, let's actually calculate. Two points, say (-1,-1) and (1,1). Each can be moved left or right by 1/2, with equal probability of each direction.

There are four possible slopes, given as y difference/x difference

• LL: (2/2)
• LR: (2/3)
• RL: (2/1)
• RR: (2/2)

That's interesting: the average slope over those four possibilities is 1.16667, which is greater than the true slope of 1. The denominator (the x-difference) averages to 2, but the non-linearity from taking the reciprocal makes the slope averages to more than 2/2 (by Jensen's inequality) -- not less, and not equal to.

Is it something special about having just two points? We can try with lots of pairs of points (note: this is lots of pairs, it isn't a sample with equal probability on $$\pm 1$$)

> LOTS<-1000
> x<-rep(c(-1,1), each=LOTS)
> z<-x+rbinom(2*LOTS, 1, .5)-1/2
> y<-x
> lm(y~x) #check

Call:
lm(formula = y ~ x)

Coefficients:
(Intercept)            x
2.329e-31    1.000e+00

> lm(y~z)

Call:
lm(formula = y ~ z)

Coefficients:
(Intercept)            z
-0.001197     0.797857


Ok, so with lots of pairs of points we get the expected dilution, so it is something about the number of points. The next step is to try different numbers of pairs: each point on the graph is an average of 1000 replicates.

There is dilution as soon as you have more than one pair, and it settles down pretty fast to the usual value.

I think this has something to do with the fixed x values; if $$x$$ were randomly drawn with 50% probability on $$\pm 1$$ things would be different. You could then have two $$x$$ values that were the same and became different only with error, so you'd have a possible horizontal line.

However, it's pretty clear that you won't understand the general problem by working with two points this way, because the results are indeed different. This has also been an example of calculations beating handwaving.

late to this sorry David but my view is regression dilution can't be appreciated well with just 2 points. If you have a decent sized sample where the x variable is measured with error then those measured x variables at the extreme will be more extreme tan their true values diluting the regression coeficiant as the y value will be affected by the true x

Maybe we can play around with this code to show its true

#regression dilution bias
np<-30
x<-rnorm(np,30,4)
y<-2.2*x + rnorm(np,0,1)
set.seed(1066)
slopes<-numeric(1000)
for(i in 1:1000)
slopes[i]<-coef(lm(y ~ rnorm(np,x,1)))[2]
summary(slopes)