24
$\begingroup$

There has been some confusion in my head about two types of estimators of the population value of Pearson correlation coefficient.

A. Fisher (1915) showed that for bivariate normal population empirical $r$ is a negatively biased estimator of $\rho$, although the bias can be of practically considerable amount only for small sample size ($n<30$). Sample $r$ underestimates $\rho$ in the sense that it is closer to $0$ than $\rho$. (Except when the latter is $0$ or $\pm 1$, for then $r$ is unbiased.) Several nearly unbiased estimators of $\rho$ has been proposed, the best one probably being Olkin and Pratt (1958) corrected $r$:

$$r_\text{unbiased} = r \left [1+\frac{1-r^2}{2(n-3)} \right ]$$

B. It is said that in regression observed $R^2$ overestimates the corresponding population R-square. Or, with simple regression, it is that $r^2$ overestimates $\rho^2$. Based on that fact, I've seen many texts saying that $r$ is positively biased relative to $\rho$, meaning absolute value: $r$ is farther from $0$ than $\rho$ (is that statement true?). The texts say it is the same problem as the over-estimation of the standard deviation parameter by its sample value. There exist many formulas to "adjust" observed $R^2$ closer to its population parameter, Wherry's (1931) $R_\text{adj}^2$ being the most well-known (but not the best). The root of such adjusted $r_\text{adj}^2$ is called shrunken $r$:

$$r_\text{shrunk} = \pm\sqrt{1-(1-r^2)\frac{n-1}{n-2}}$$

Present are two different estimators of $\rho$. Very different: the first one inflates $r$, the second deflates $r$. How to reconcile them? Where to use/report one and where - the other?

In particular, can it be true that the "shrunken" estimator is (nearly) unbiased too, like the "unbiased" one, but only in the different context - in the asymmetrical context of regression. For, in OLS regression we consider the values of one side (the predictor) as fixed, attending without random error from sample to sample? (And to add here, regression does not need bivariate normality.)

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ I wonder if this just comes down to something based on Jensen's inequality. That, and bivariate normality is probably a bad assumption in most cases. $\endgroup$
    – shadowtalker
    Commented Apr 29, 2015 at 3:35
  • 2
    $\begingroup$ Also, my understanding of the issue in B. is that regression $r^2$ is an overestimate because the regression fit can be improved arbitrarily by adding predictors. That doesn't sound to me like the same issue as in A. $\endgroup$
    – shadowtalker
    Commented Apr 29, 2015 at 3:48
  • 1
    $\begingroup$ Is it actually true that $r^2$ is a positively biased estimate of $\rho^2$ for all values of $\rho$? For the bivariate normal distribution this does not seem to be the case for $\rho$ large enough. $\endgroup$
    – NRH
    Commented May 4, 2015 at 21:48
  • 1
    $\begingroup$ Can bias go in the opposite direction for the square of an estimator? For example, with a simpler estimator, can it be shown that $E[\hat{\theta}-\theta] < 0 < E[\hat{\theta}^2-\theta^2]$ for some ranges of $\theta$? I think this would be difficult to do if $\theta = \rho$, but perhaps a simpler example could be worked out. $\endgroup$
    – Anthony
    Commented May 4, 2015 at 23:09
  • $\begingroup$ The link to Fisher (1915) is dead, here is the new one at Duke: www2.stat.duke.edu/courses/Spring05/sta215/lec/Fish1915.pdf A more stable one: jstor.org/stable/2331838 $\endgroup$
    – user384760
    Commented Apr 16, 2023 at 18:47

2 Answers 2

Reset to default
1
$\begingroup$

Regarding the bias in the correlation: When sample sizes are small enough for bias to have any practical significance (e.g., the n < 30 you suggested), then bias is likely to be the least of your worries, because inaccuracy is terrible.

Regarding the bias of R2 in multiple regression, there are many different adjustments that pertain to unbiased population estimation vs. unbiased estimation in an independent sample of equal size. See Yin, P. & Fan, X. (2001). Estimating R2 shrinkage in multiple regression: A comparison of analytical methods. The Journal of Experimental Education, 69, 203-224.

Modern-day regression methods also address the shrinkage of regression coefficients as well as R2 as a consequence -- e.g., the elastic net with k-fold cross validation, see http://web.stanford.edu/~hastie/Papers/elasticnet.pdf.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I don't know if this really answers the question $\endgroup$
    – shadowtalker
    Commented Apr 29, 2015 at 3:35
1
$\begingroup$

I think the answer is in the context of simple regression and multiple regression. In simple regression with one IV and one DV, the R sq is not positively biased, and in-fact may be negatively biased given r is negatively biased. But in multiple regression with several IV's which may be correlated themselves, R sq may be positively biased because of any "suppression" that may be happening. Thus, my take is that observed R2 overestimates the corresponding population R-square, but only in multiple regression

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ R sq is not positively biased, and in-fact may be negatively biased Interesting. Can you show it or give a reference? - In bivariate normal population, can observed sample Rsq statistic be negatively biased estimator? $\endgroup$
    – ttnphns
    Commented Nov 25, 2016 at 21:29
  • $\begingroup$ I think you are wrong. Could you give a reference to back up your claim? $\endgroup$
    – Richard Hardy
    Commented Nov 27, 2016 at 16:27
  • $\begingroup$ Sorry, but this was more of a thought exercise, so I have no reference. $\endgroup$
    – Dingus
    Commented Nov 29, 2016 at 19:41
  • 1
    $\begingroup$ I was going off of Comment A above, where Fischer showed that in a bivariate normal situation, r is a negatively biased estimator of rho. If that is the case would it not follow that R sq is also negatively biased? $\endgroup$
    – Dingus
    Commented Nov 29, 2016 at 19:43
  • $\begingroup$ Perhaps this will aid in the conversation digitalcommons.unf.edu/cgi/… $\endgroup$
    – Dingus
    Commented Nov 29, 2016 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.