2
$\begingroup$

Many similar questions here, but this question combines two threads: relationship between correlation and regression, and correlation between a binary and a continuous variable.

  1. Since the relationship between a binary variable and a continuous variable is not linear, is Pearson correlation ever an appropriate measure of whether these two variables move together? Several sources mention that Point Biserial is equivalent to Pearson correlation in case one variable is binary and the other continuous. So is point Biserial correlation identical to Pearson correlation in this case? Is Point Biserial more appropriate/the most appropriate?
  2. Is it still the case that R-squared from OLS equals the square of Pearson correlation, if the dependent variable is binary and the independent variable is continuous? Why?
  3. Is there some type of regression between these variables for which R-squared is the square of Point Biserial correlation?
$\endgroup$
3
  • 3
    $\begingroup$ One crucial difference is that correlation is 'symmetrical': $Cor(X,Y) = Cor(Y,X).$ However, regression is not: Regression line of $Y$ on $X$ is not at all the same as regression line of $X$ on $Y.$ $\endgroup$
    – BruceET
    Commented Jan 24, 2019 at 22:56
  • $\begingroup$ (1) The relationship between a binary variable and a continuous variable is always linear. (2) Because $R^2=r^2$ is a mathematical result and math doesn't care how you think of a variable, a fortiori the answer is yes. Thus, answers should focus on the remaining question (3). $\endgroup$
    – whuber
    Commented Jul 8, 2022 at 13:44
  • $\begingroup$ Why must the relationship between a binary variable and a continuous variable be nonlinear? (Are you thinking of a logistic regression?) $\endgroup$
    – Dave
    Commented Jul 8, 2022 at 14:52

1 Answer 1

3
$\begingroup$

As I remarked in a comment, the interesting question is (3): how might $R^2$ and the point biserial correlation (p.b.c.) coefficient be related? In fact, all three questions will be answered in the following analysis.

We need formulas for the terms that go into computing $R^2$ and the p.b.c. coefficient, but since this focuses on a pair of variables $(X,Y)$ in which (say) $Y$ is binary, these formulas ought to be simple. As usual, code $Y$ with values $0$ and $1.$ (The particular values don't matter, because correlations do not change when values are shifted or rescaled, but this set of values is conventional and easy to work with.)

Following Wikipedia, adopt the following notation:

  • $n_0$ is the number of observations for which $Y=0$ and $n_1$ is the count of observations for which $Y=1.$

  • $n = n_0 + n_1$ is the total number of observations.

  • $\bar X = (X_1 + X_2 + \cdots + X_n)/n$ is the mean of the $X$ data.

  • $\bar Y = (n_0\times 0 + n_1\times 1)/n = n_1/n$ is the mean of the $Y$ data.

  • Similarly, $M_0$ is the mean of the $X$ data corresponding to values where $Y=0$ and $M_1$ is the mean of the $X$ data for $Y=1.$

    • Notice that $n_0 M_0 + n_1 M_1 = n \bar X$ because both expressions equal the sum of the $X$ data.

    • The mean of an empty list of values is undefined.

  • $n s^2 = (X_1-\bar X)^2 + (X_2-\bar X)^2 + \cdots + (X_n-\bar X)^n$ makes $s^2$ the (empirical) variance of the $X$ data (not the unbiased estimator!).

    • I will write $s^2 = \operatorname{Var}(X)$ and $s = \operatorname{SD}(X)$ as a shorthand.

    • The residuals are the values relative to their means: $X_i-\bar X$ for the $X$ data and $Y_i - \bar Y$ for the $Y$ data.

  • The empirical variance of the $Y$ data is $$\operatorname{Var}(Y) = \frac{1}{n}\left[n_0(0 - \bar Y)^2 + n_1(1 - \bar Y)^2\right] = \frac{n_0n_1}{n^2}.$$

  • The empirical covariance of the $(X,Y)$ data is the mean product of the paired residuals. In $n_0$ of the cases the $Y$ residual is $0-\bar Y=-n_1/n$ while in the remaining $n_1$ of the cases the $Y$ residual is $1-\bar Y = 1 - n_1/n = n_0/n.$ Consequently, $$\begin{aligned}\operatorname{Cov}(X,Y) &= \frac{1}{n}\left[ n_0 \frac{-n_1}{n}(M_0 - \bar X) + n_1 \frac{n_0}{n}(M_1 - \bar X)\right] = \frac{n_0n_1}{n^2}\left[M_1 - M_0\right].\end{aligned}$$

The point biserial correlation coefficient is defined to be

$$r_{pb} = \frac{M_1 - M_0}{s}\sqrt{\frac{n_0n_1}{n^2}} = \frac{\frac{n_0n_1}{n^2}\left[M_1-M_0\right]}{s\,\sqrt{n_0n_1/n^2}} = \frac{\operatorname{Cov}(X,Y)}{\operatorname{SD}(X)\operatorname{SD}(Y)} = \rho(X,Y)$$

where $\rho$ is the Pearson correlation coefficient.

Because $R^2(X,Y) = \rho(X,Y)^2$ (see the formulas at the end of https://stats.stackexchange.com/a/579154/919, for instance),

$$r_{pb} = \rho\quad \text{ and }\quad R^2 = r_{pb}^2 = \rho^2.$$

Incidentally, as you may check, whenever all the data are in one group ($Y=0$ only or $Y=1$ only) all three quantities--$R^2,$ $r_{pb},$ and $\rho$--are undefined.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.