5
$\begingroup$

I am trying to understand relation between bias and R-squared value in linear regression.

High bias means that the model is underfit. By this I am assuming that the R-square d will be less. So my doubt is, can we say that bias and R-squared are inversely proportional?

Also is there any such relation between variance and R-Squared value?

Improve this question
$\endgroup$
3
  • $\begingroup$ Do you understand how these relate to MSE or SSE? $\endgroup$
    – Dave
    Commented Jan 10, 2020 at 18:06
  • 3
    $\begingroup$ Please explain what you mean by "bias" and what method of linear regression you are using. The reason we need this is that according to a standard statistical meaning (that the expectation of an estimate equals its estimand), standard theorems about ordinary least squares regression state that all parameter estimates are unbiased, whence there is no relationship at all between that sense of "bias" and $R^2.$ $\endgroup$
    – whuber ♦
    Commented Jan 10, 2020 at 18:25
  • $\begingroup$ By "bias" I mean the difference between average prediction and the actual value, which is used to evaluate a linear regression model. multiple linear regression can be taken for example. Got your point. I was just going through bias and R-square to understand what they are and I thought if there is any relation between them. $\endgroup$
    – AnkitD
    Commented Jan 12, 2020 at 16:53

1 Answer 1

Reset to default
0
$\begingroup$

It depends on how you define $R^2$, and there are several legitimate definitions, with the two below being the most reasonable to me.

$$ R^2 =\left(\text{corr}\left(\hat y, y\right)\right)^2\\ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

$R^2 =\left(\text{corr}\left(\hat y, y\right)\right)^2$ will not care about bias. If you predict one unit too high every time, the correlation is perfect. Quoting an answer of mine from Data Science, for any real $a$ and positive $b$, $ \left(\text{corr}\left(\hat y, y\right)\right) = \left(\text{corr}\left(a + b\hat y, y\right)\right) $. In fact, for nonzero $b$, $ \left(\text{corr}\left(\hat y, y\right)\right)^2 = \left(\text{corr}\left(a + b\hat y, y\right)\right)^2 $.

Concretely, if your true values are $y=(1,2,4,6,5)$, $\hat y=(12, 13, 15, 17, 16)$ makes for terrible predictions but does have a perfect correlation with $y$. (In this example, $a=11$ and $b=1$.)

For the second formula, the numerator is just a function of the mean squared error. Since the denominator is a function of the data and not of any particular model, regard the denominator as a constant (which is is for any given data set). Then the second formula is just a decreasing function of the mean squared error. Since, all else equal, mean squared error increases when bias magnitude increases ($MSE = \text{bias}^2 + \text{var}$), you can regard that equation for $R^2$ as moving in the opposite direction of bias magnitude, yes. As bias magnitude decreases, $R^2$ increases, and as bias magnitude increases, $R^2$ decreases (assuming equal variance in both cases). This makes sense to me. Holding the variance equal, higher bias magnitude means a worse fit, which should correspond to a lower $R^2$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ To get a handle on the whole story, I would encourage you to explicitly take a derivative of $R^2$ with respect to $\text{bias}$. $\endgroup$
    – Dave
    Commented Mar 23, 2023 at 20:53
  • $\begingroup$ I doubt about the relation between bias and any fitting mesure, so between bias and R^2 even defined in your second form. Consider that BVT is defined on the expected generalization error, an out of sample measure, while the $R^2$ is an in sample one. $\endgroup$
    – markowitz
    Commented May 20 at 6:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.