Linear regression formula interpretation in term of correlation I have been taught that the optimal coefficients (in the MMSE sense) may be obtained by looking where the gradient of the associated loss is zero :
With $ L(D,\beta) = ||X\beta-Y||^2 $ :
$$ \frac{\partial L(D,\beta)}{\partial \beta} = 0 <=> \hat\beta = (X^TX)^{-1} X^Ty $$
I applied this formula religiously since then. However I am now trying to interpret it in terms of $(X^TX)^{-1}$ and $X^Ty$. They appear respectively linked to the invert of the correlation matrix of the $X_i$ and a vector showing the correlation of the $X_i$ with $y$ (modulo centering and normalisation).
How would you explain simply that applying the invert of the correlation matrice of the variables, to the vector of correlation between the variable and the output will - modulo centering and normalisation - minimise the mean squared loss and yield appropriate coefficients for linear regression ?
 A: As a newby here, I cannot add a comment, so my answer will be more like an elaborated comment. 
Think of a single variable in $X$ and $y$, then your estimate $\hat{\beta} = \frac{cov(x,y)}{Var(x)}$. 
This captures covariation between $x$ and $y$ standardized in terms of variance of $x$. And as you have said, the estimate $\beta$ is very much related to the correlation coefficient, but notice that it does not equal to it, which is given by $\rho = \frac{cov(x,y)}{std(x) std(y)}$. In fact, they capture the same effect, namely the linear covariation between the variables, but they are standardized by the different values.
Now, in your question the matrix $X^TX$ is not a correlation matrix, but rather related to the covariance matrix of $X$. And also $X^Ty$ captures covariation effect between all $X$'s and y. So in the multivariate $X$, your estimates $\hat \beta$ would now be related to the partial correlations rather than ordinary correlations between all $X$ with $y$. That is, they capture similar effects, but again they would be standardized by the variance of $X$. And for the latter statement, I just refer to the answer in this question.
Hope this helps a bit.
