Link between linear regression and correlation

Question

I happen to do a lot of linear regression, the typical use case is predicting $Y$ with a few predictors say $X = [X_1;X_2;X_3]$. So we we find $\hat{\beta}$ such that $$\hat{\beta} = \underset{\beta}{argmin} ||Y-X.\beta||^2$$ But actually what I am more interested in in finding the linear estimation of $Y$ such that $$corr(Y,\hat{Y}) = corr(Y,X.\hat{\beta})$$ is maximal.

I am bothered by the fact that for a given $\beta_0$, $corr(Y,X.a.\beta_0)$ is the same for all $a$, questions are then

Is the optimal linear estimator of $\beta$ in the least square sense also optimal in my correlation term, and if yes should I be bothered that the optimal $\beta$ is not unique ?

Answer 1

Think about $R^2$ of linear regression:

$R^2=1-\frac{\|Y-X\hat\beta_{OLS}\|^2}{\|Y-\bar Y\|^2}=[Corr(Y,X\hat\beta_{OLS})]^2$

Since denominator does not depend on $\hat\beta$ and LS minimizes the numerator the "result follows".

---

You correctly mentioned that the correlation coefficient is invariant under linear transformations (which preserve sign). It means that you cannot use correlation to recover the underlying parameters.

Consider the next univariate model:

$Y_i=X_i\beta+u_i$

For any $b\not=0, |Corr(Y_i. X_ib)|=|Corr(Y_i, X_i)|$

So, it meaningless to maximize sample covariance w.r.t to $b$.

On the other hand, $R^2(b)\le Corr(Y_i, X_ib)^2$ with equality for $b=b_{OLS}$. In this sense LS yields the optimal estimator.

Answer 2

The equality $1-\frac{\|Y-X\beta\|^2}{\|Y-\bar Y\|^2}=[Corr(Y,X\beta)]^2$ doesn't hold in general. Therefore the argument in d.k.o.'s answer is incorrect.

Besides, the equality holds for the choice $\beta=\hat\beta_{OLS}$ only if there is an intercept in the regression, i.e., $$\hat\alpha_{OLS}, \hat\beta_{OLS}= \underset{\alpha,\beta}{argmin} ||Y-\alpha 1-X\beta||^2$$

The correct proof, assuming an intercept, is as follows:

$$corr(Y,X\beta)=\frac{cov(Y,X)'\beta}{(\beta'var(X)\beta)^{1/2} var(Y)^{1/2}}\leq \frac{(cov(Y,X)'var(X)^{-1}cov(Y,X))^{1/2}}{var(Y)^{1/2}}$$

$$=\frac{var(X\hat\beta_{OLS})^{1/2}}{var(Y)^{1/2}}=\frac{cov(Y,X\hat\beta_{OLS})}{var(Y)^{1/2}var(X\hat\beta_{OLS})^{1/2}}=corr(Y,X\hat\beta_{OLS}),$$

where the inequality is by Cauchy-Schwarz, the second equality is because $\hat \beta_{OLS}=var(X)^{-1}cov(Y,X)$, and the third equality is because $cov(Y-X\hat\beta_{OLS},X\hat\beta_{OLS})=0$.

Therefore, $\hat\beta_{OLS}$ solves the problem $\max_\beta corr(Y,X\beta)$.

If $\|\hat\beta_{OLS}\|\neq0$, then the solution is unique up to scale, i.e., $\hat{\beta}$ solves the problem $\max_\beta corr(Y,X\beta)$ if and only if $\hat{\beta}=\lambda \hat\beta_{OLS}$ for some scalar $\lambda\neq 0$ (the Cauchy-Schwarz inequality becomes an equality).