3
$\begingroup$

I have been taught that the optimal coefficients (in the MMSE sense) may be obtained by looking where the gradient of the associated loss is zero :

With $ L(D,\beta) = ||X\beta-Y||^2 $ :

$$ \frac{\partial L(D,\beta)}{\partial \beta} = 0 <=> \hat\beta = (X^TX)^{-1} X^Ty $$

I applied this formula religiously since then. However I am now trying to interpret it in terms of $(X^TX)^{-1}$ and $X^Ty$. They appear respectively linked to the invert of the correlation matrix of the $X_i$ and a vector showing the correlation of the $X_i$ with $y$ (modulo centering and normalisation).

How would you explain simply that applying the invert of the correlation matrice of the variables, to the vector of correlation between the variable and the output will - modulo centering and normalisation - minimise the mean squared loss and yield appropriate coefficients for linear regression ?

Improve this question
$\endgroup$
5
  • $\begingroup$ I think you are over complicating this. When you take the partial derivative and set it equal to 0, you are minimizing your residuals. Think about what you are doing instead of interpreting the form of the partial derivate. $\endgroup$
    – M Waz
    Commented Oct 4, 2019 at 17:23
  • 1
    $\begingroup$ We have several threads providing geometric explanations. Given the intimate connection between the Euclidean dot product and correlation, all of these offer answers to your question: stats.stackexchange.com/questions/54943, stats.stackexchange.com/questions/97746, stats.stackexchange.com/questions/123651, and stats.stackexchange.com/questions/208175. $\endgroup$
    – whuber ♦
    Commented Oct 4, 2019 at 17:44
  • $\begingroup$ @whuber : your links are really interesting. Especially the first one where the coefficient equation shows up as $ x^T(y-x\hat\beta) $ following a geometric reasonning. It helps understand where the $(X^TX)^{-1}$ term come from. However I am not sure it answers my question which was about an interpretation in terms of correlation. $\endgroup$
    – Lucas Morin
    Commented Oct 4, 2019 at 18:25
  • $\begingroup$ The correlation of two centered vectors $x$ and $y$ is $x\cdot y / (|x||y|).$ This expression or the closely related one $x\cdot y$ shows up in most of the linked threads. $\endgroup$
    – whuber ♦
    Commented Oct 4, 2019 at 18:37
  • $\begingroup$ Yes, this is the basis of my question. The linear regression formula, which make sense geometrically, as the projection of y on the space generated by the Xis, contain terms that can be rewritten as correlation matrices / vectors. Once you consider the formula in terms of correlations, how does it make sense ? $\endgroup$
    – Lucas Morin
    Commented Oct 5, 2019 at 9:20

1 Answer 1

Reset to default
1
$\begingroup$

As a newby here, I cannot add a comment, so my answer will be more like an elaborated comment.

Think of a single variable in $X$ and $y$, then your estimate $\hat{\beta} = \frac{cov(x,y)}{Var(x)}$.

This captures covariation between $x$ and $y$ standardized in terms of variance of $x$. And as you have said, the estimate $\beta$ is very much related to the correlation coefficient, but notice that it does not equal to it, which is given by $\rho = \frac{cov(x,y)}{std(x) std(y)}$. In fact, they capture the same effect, namely the linear covariation between the variables, but they are standardized by the different values.

Now, in your question the matrix $X^TX$ is not a correlation matrix, but rather related to the covariance matrix of $X$. And also $X^Ty$ captures covariation effect between all $X$'s and y. So in the multivariate $X$, your estimates $\hat \beta$ would now be related to the partial correlations rather than ordinary correlations between all $X$ with $y$. That is, they capture similar effects, but again they would be standardized by the variance of $X$. And for the latter statement, I just refer to the answer in this question.

Hope this helps a bit.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.