4
$\begingroup$

I've got a question about errors in variables. So, if I run a standard linear regression to estimate b in y = a + bx, but my independent variable x is measured with error, x = x' + x" where x' is the true value, then my estimate of b will be wrong:

$E\hat{b} = b (1 - \frac{var(x^")}{var(x') + var(x^")})$

I hope I have got that right.

My question is, how will the R squared of this regression be related to the true R squared, i.e. the proportion of variance in y explained by the true value x'?

Put it another way: suppose I have an R squared and a coefficient from a given regression of a variable measured with error; and I know the R squared that would be achieved if I could measure the variable perfectly; can I deduce an estimate of the true regression coefficient b?

$\endgroup$
4
+50
$\begingroup$

Yes, it's possible, but it requires a dearth of simpifying assumptions, not always likely to hold in practice. Let's assume the following model

$$y=\alpha+\beta x^*+\epsilon$$

As usual, we assume $E[x^*\epsilon]=0$. Since you didn't mention measurement error for $y$, I won't include it. However, we do have measurement error on the predictor, i.e., we don't observe $x^*$ but $x$, which is related to $x^*$ by the model

$$x=x^*+\eta$$

Now we assume that $x^*$ and $\eta$ are independent, that $E[\eta]=0$ and that $E[\eta^2]=\sigma^2_{\eta}$ is known. Note that all this is quite restrictive! For most measurement instruments, the RMSE isn't constant across all the scale (i.e., for each value of the measured variable inside the non-saturated range), and even if it were, the RMSE wouldn't be known exactly. However, if the instrument has been calibrated multiple times by a certified laboratory, we may assume that $\sigma^2_{\eta}$ is reasonably well-known. Also, there are instruments such that, when taking measurements of $x^*$ sufficiently far from the ends of the scale, the RMSE stays the same, whatever the value of $x^*$ in this range. Thus, there are (few) real situations where these hypotheses are reasonable. Finally, we assume that $\epsilon$ and $\eta$ are independent. Note also that this model leads to

$$y=\alpha+\beta (x-\eta)+\epsilon=\alpha+\beta x+\epsilon-\beta\eta=\alpha+\beta x+u$$

Enough of the background: when we observe $\{(x_i,y_i)\}_{i=1,\dots,N}$, the estimate of $R^2$ is

$$\hat{R}^2=1-\frac{\sum_{i=1}^N(\hat{\alpha}+\hat{\beta} x_i-y_i)^2}{\sum_{i=1}^N(y_i-\bar{y})^2}$$

Now we're in for a bit of calculus:

  1. $$\text{plim}\frac{N}{\sum_{i=1}^N(y_i-\bar{y})^2}=\frac{1}{\sigma^2_y}$$

  2. $$\text{plim}\hat{\beta}=\text{plim}\frac{\sum_{i=1}^N(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^N(x_i-\bar{x})^2}=\frac{\sigma_{xy}}{\sigma^2_x}=\frac{\text{Cov}[(x^*+\eta)(\alpha+\beta x^* +\epsilon)]}{\sigma^2_x}=0+\beta\frac{\sigma^2_{x^*}}{\sigma^2_x}+\beta\frac{\text{Cov}[\eta x^*]}{\sigma^2_x}+\beta\frac{\text{Cov}[\eta\epsilon]}{\sigma^2_x}=\beta\frac{\sigma^2_*}{\sigma^2_x}$$ where in the last step we used the fact that $\text{Cov}[\eta x^*]=0$ (since $x^*$ and $\eta$ are independent and $E[\eta]=0$) and we used the hypothesis that $\eta$ and $\epsilon$ are independent with 0 mean. We also used the short-hand notation $\sigma^2_*$ for $\sigma^2_{x^*}$. Finally, since $$\sigma^2_x=\text{Var}[(x^*+\eta)^2]=\sigma^2_*+2\text{Cov}[\eta x^*]+\sigma^2_{\eta}=\sigma^2_*+\sigma^2_{\eta}$$ we get the well-known expression $$\text{plim}\hat{\beta}=\beta\frac{\sigma^2_*}{\sigma^2_*+\sigma^2_{\eta}}$$

  3. Using the fact that $\hat{\alpha}=\bar{y}-\hat{\beta}\bar{x}$, we can simplify the numerator in the expression of $\hat{R}^2$: $$\sum_{i=1}^N(\hat{\alpha}+\hat{\beta} x_i-y_i)^2=\sum_{i=1}^N(\bar{y}-\hat{\beta}\bar{x}+\hat{\beta} x_i-y_i)^2=\sum_{i=1}^N(\hat{\beta}(x_i-\bar{x}) -(y_i-\bar{y}))^2$$ Now, we can use 2. to compute the following probability limit: $$\text{plim}\frac{\sum_{i=1}^N(\hat{\beta}(x_i-\bar{x})-(y_i-\bar{y}))^2}{N}=\text{plim}\frac{\sum_{i=1}^N \hat{\beta}^2(x_i-\bar{x})^2+(y_i-\bar{y})^2-2\hat{\beta}(x_i-\bar{x})(y_i-\bar{y})}{N}=\beta^2\frac{\sigma^4_*}{\sigma^4_x}\sigma^2_x+\sigma^2_y-2\beta\frac{\sigma^2_*}{\sigma^2_x}\sigma_{xy}$$

In 2. we've already seen that $\sigma_{xy}=\beta\sigma^2_*$, thus

$$\beta^2\frac{\sigma^4_*}{\sigma^4_x}\sigma^2_x+\sigma^2_y-2\beta\frac{\sigma^2_*}{\sigma^2_x}\sigma_{xy}=\beta^2\frac{\sigma^4_*}{\sigma^2_x}+\sigma^2_y-2\beta^2\frac{\sigma^4_*}{\sigma^2_x}=\sigma^2_y-\beta^2\frac{\sigma^4_*}{\sigma^2_x}$$

  1. Putting this all together:

$$\text{plim}\hat{R}^2=1-\frac{\sigma^2_y}{\sigma^2_y}+\beta^2\frac{\sigma^4_*}{\sigma^2_x\sigma^2_y}=\beta^2\frac{\sigma^4_*\sigma^2_*}{\sigma^2_x\sigma^2_*\sigma^2_y}=R^2\frac{\sigma^2_*}{\sigma^2_x}$$

because, only for simple linear regression,

$$R^2=r^2=\beta^2\frac{\sigma^4_*}{\sigma^2_*\sigma^2_y}$$

Thus we (finally!!) conclude that

$$\text{plim}\hat{R}^2=R^2\frac{\sigma^2_x-\sigma^2_\eta}{\sigma^2_x}$$

which is the desired relationship between the R squared of our regression with errors in the predictor, and the true $R^2$. Note that, as well-known, in the classic error-in-variables model the OLS estimator of $R^2$ converges to a limit which is always smaller than the true $R^2$. Finally, if an estimate of $\sigma^2_{\eta}$ is known (by independent calibration of the instrument: you cannot get it from the same data used for the regression), then, since the data $\{(x_i,y_i)\}_{i=1,\dots,N}$ allow an estimate of ${\sigma^2_x}$ and ${\sigma^2_y}$, you see that we can get an estimate of $\beta$.

$\endgroup$
  • 1
    $\begingroup$ This looks really solid. Just to double check I've understood correctly: the estimated $\hat{R}^2$, and the known $R^2$, provide an estimate of $\beta^2 \sigma^2_\eta / \sigma^2_y $ . But we can't get $ \beta $ itself out without also knowing the variance of the measurement error. $\endgroup$ – dash2 Oct 9 '16 at 21:47
  • $\begingroup$ Yes, exactly. Of course, this isn't the only possible errors-in-variables model: for example, you could assume that $\sigma_{\epsilon}/\sigma_{\eta}$ is known (Deming regression) or that the reliability $\lambda=\sigma^2_*/(\sigma^2_*+\sigma^2_{\eta}) $ is known , where $\sigma^2_*$ is the variance of the latent predictor (it's immediate to show that this last case is equivalent to my model where $\sigma_{\eta}$ is assumed to be known). Anyway, all classical errors-in-variables models are similar, in that quite strong assumptions are made. $\endgroup$ – DeltaIV Oct 9 '16 at 22:22
  • $\begingroup$ Oh, sh... I just now realized I used the true $\alpha$ and $\beta$ in my formulas, while of course I must use the estimators, which are biased in this context, so they don't converge to the "true" values in the $\text{plim}$. I was fooled by the fact that the result looked good: it's a well-known fact that error-in-variables leads to an $R^2$ which is smaller than the "true" value, unless you use an instrumental variable to compute it. No worries, though - I'm fixing this right away. Let me know if you like the new version. $\endgroup$ – DeltaIV Oct 10 '16 at 11:33
  • $\begingroup$ It looks the same so far... $\endgroup$ – dash2 Oct 10 '16 at 12:43
  • $\begingroup$ @dash2 we're a bit impatient, it seems ;) here it is. $\endgroup$ – DeltaIV Oct 10 '16 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.