# How do errors in variables affect the R2?

I've got a question about errors in variables. So, if I run a standard linear regression to estimate b in y = a + bx, but my independent variable x is measured with error, x = x' + x" where x' is the true value, then my estimate of b will be wrong:

$E\hat{b} = b (1 - \frac{var(x^")}{var(x') + var(x^")})$

I hope I have got that right.

My question is, how will the R squared of this regression be related to the true R squared, i.e. the proportion of variance in y explained by the true value x'?

Put it another way: suppose I have an R squared and a coefficient from a given regression of a variable measured with error; and I know the R squared that would be achieved if I could measure the variable perfectly; can I deduce an estimate of the true regression coefficient b?

Yes, it's possible, but it requires a dearth of simpifying assumptions, not always likely to hold in practice. Let's assume the following model

$$y=\alpha+\beta x^*+\epsilon$$

As usual, we assume $E[x^*\epsilon]=0$. Since you didn't mention measurement error for $y$, I won't include it. However, we do have measurement error on the predictor, i.e., we don't observe $x^*$ but $x$, which is related to $x^*$ by the model

$$x=x^*+\eta$$

Now we assume that $x^*$ and $\eta$ are independent, that $E[\eta]=0$ and that $E[\eta^2]=\sigma^2_{\eta}$ is known. Note that all this is quite restrictive! For most measurement instruments, the RMSE isn't constant across all the scale (i.e., for each value of the measured variable inside the non-saturated range), and even if it were, the RMSE wouldn't be known exactly. However, if the instrument has been calibrated multiple times by a certified laboratory, we may assume that $\sigma^2_{\eta}$ is reasonably well-known. Also, there are instruments such that, when taking measurements of $x^*$ sufficiently far from the ends of the scale, the RMSE stays the same, whatever the value of $x^*$ in this range. Thus, there are (few) real situations where these hypotheses are reasonable. Finally, we assume that $\epsilon$ and $\eta$ are independent. Note also that this model leads to

$$y=\alpha+\beta (x-\eta)+\epsilon=\alpha+\beta x+\epsilon-\beta\eta=\alpha+\beta x+u$$

Enough of the background: when we observe $\{(x_i,y_i)\}_{i=1,\dots,N}$, the estimate of $R^2$ is

$$\hat{R}^2=1-\frac{\sum_{i=1}^N(\hat{\alpha}+\hat{\beta} x_i-y_i)^2}{\sum_{i=1}^N(y_i-\bar{y})^2}$$

Now we're in for a bit of calculus:

1. $$\text{plim}\frac{N}{\sum_{i=1}^N(y_i-\bar{y})^2}=\frac{1}{\sigma^2_y}$$

2. $$\text{plim}\hat{\beta}=\text{plim}\frac{\sum_{i=1}^N(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^N(x_i-\bar{x})^2}=\frac{\sigma_{xy}}{\sigma^2_x}=\frac{\text{Cov}[(x^*+\eta)(\alpha+\beta x^* +\epsilon)]}{\sigma^2_x}=0+\beta\frac{\sigma^2_{x^*}}{\sigma^2_x}+\beta\frac{\text{Cov}[\eta x^*]}{\sigma^2_x}+\beta\frac{\text{Cov}[\eta\epsilon]}{\sigma^2_x}=\beta\frac{\sigma^2_*}{\sigma^2_x}$$ where in the last step we used the fact that $\text{Cov}[\eta x^*]=0$ (since $x^*$ and $\eta$ are independent and $E[\eta]=0$) and we used the hypothesis that $\eta$ and $\epsilon$ are independent with 0 mean. We also used the short-hand notation $\sigma^2_*$ for $\sigma^2_{x^*}$. Finally, since $$\sigma^2_x=\text{Var}[(x^*+\eta)^2]=\sigma^2_*+2\text{Cov}[\eta x^*]+\sigma^2_{\eta}=\sigma^2_*+\sigma^2_{\eta}$$ we get the well-known expression $$\text{plim}\hat{\beta}=\beta\frac{\sigma^2_*}{\sigma^2_*+\sigma^2_{\eta}}$$

3. Using the fact that $\hat{\alpha}=\bar{y}-\hat{\beta}\bar{x}$, we can simplify the numerator in the expression of $\hat{R}^2$: $$\sum_{i=1}^N(\hat{\alpha}+\hat{\beta} x_i-y_i)^2=\sum_{i=1}^N(\bar{y}-\hat{\beta}\bar{x}+\hat{\beta} x_i-y_i)^2=\sum_{i=1}^N(\hat{\beta}(x_i-\bar{x}) -(y_i-\bar{y}))^2$$ Now, we can use 2. to compute the following probability limit: $$\text{plim}\frac{\sum_{i=1}^N(\hat{\beta}(x_i-\bar{x})-(y_i-\bar{y}))^2}{N}=\text{plim}\frac{\sum_{i=1}^N \hat{\beta}^2(x_i-\bar{x})^2+(y_i-\bar{y})^2-2\hat{\beta}(x_i-\bar{x})(y_i-\bar{y})}{N}=\beta^2\frac{\sigma^4_*}{\sigma^4_x}\sigma^2_x+\sigma^2_y-2\beta\frac{\sigma^2_*}{\sigma^2_x}\sigma_{xy}$$

In 2. we've already seen that $\sigma_{xy}=\beta\sigma^2_*$, thus

$$\beta^2\frac{\sigma^4_*}{\sigma^4_x}\sigma^2_x+\sigma^2_y-2\beta\frac{\sigma^2_*}{\sigma^2_x}\sigma_{xy}=\beta^2\frac{\sigma^4_*}{\sigma^2_x}+\sigma^2_y-2\beta^2\frac{\sigma^4_*}{\sigma^2_x}=\sigma^2_y-\beta^2\frac{\sigma^4_*}{\sigma^2_x}$$

1. Putting this all together:

$$\text{plim}\hat{R}^2=1-\frac{\sigma^2_y}{\sigma^2_y}+\beta^2\frac{\sigma^4_*}{\sigma^2_x\sigma^2_y}=\beta^2\frac{\sigma^4_*\sigma^2_*}{\sigma^2_x\sigma^2_*\sigma^2_y}=R^2\frac{\sigma^2_*}{\sigma^2_x}$$

because, only for simple linear regression,

$$R^2=r^2=\beta^2\frac{\sigma^4_*}{\sigma^2_*\sigma^2_y}$$

Thus we (finally!!) conclude that

$$\text{plim}\hat{R}^2=R^2\frac{\sigma^2_x-\sigma^2_\eta}{\sigma^2_x}$$

which is the desired relationship between the R squared of our regression with errors in the predictor, and the true $R^2$. Note that, as well-known, in the classic error-in-variables model the OLS estimator of $R^2$ converges to a limit which is always smaller than the true $R^2$. Finally, if an estimate of $\sigma^2_{\eta}$ is known (by independent calibration of the instrument: you cannot get it from the same data used for the regression), then, since the data $\{(x_i,y_i)\}_{i=1,\dots,N}$ allow an estimate of ${\sigma^2_x}$ and ${\sigma^2_y}$, you see that we can get an estimate of $\beta$.

• This looks really solid. Just to double check I've understood correctly: the estimated $\hat{R}^2$, and the known $R^2$, provide an estimate of $\beta^2 \sigma^2_\eta / \sigma^2_y$ . But we can't get $\beta$ itself out without also knowing the variance of the measurement error. – dash2 Oct 9 '16 at 21:47
• Yes, exactly. Of course, this isn't the only possible errors-in-variables model: for example, you could assume that $\sigma_{\epsilon}/\sigma_{\eta}$ is known (Deming regression) or that the reliability $\lambda=\sigma^2_*/(\sigma^2_*+\sigma^2_{\eta})$ is known , where $\sigma^2_*$ is the variance of the latent predictor (it's immediate to show that this last case is equivalent to my model where $\sigma_{\eta}$ is assumed to be known). Anyway, all classical errors-in-variables models are similar, in that quite strong assumptions are made. – DeltaIV Oct 9 '16 at 22:22
• Oh, sh... I just now realized I used the true $\alpha$ and $\beta$ in my formulas, while of course I must use the estimators, which are biased in this context, so they don't converge to the "true" values in the $\text{plim}$. I was fooled by the fact that the result looked good: it's a well-known fact that error-in-variables leads to an $R^2$ which is smaller than the "true" value, unless you use an instrumental variable to compute it. No worries, though - I'm fixing this right away. Let me know if you like the new version. – DeltaIV Oct 10 '16 at 11:33
• It looks the same so far... – dash2 Oct 10 '16 at 12:43
• @dash2 we're a bit impatient, it seems ;) here it is. – DeltaIV Oct 10 '16 at 14:33