I would like to understand better the closed-form solution of the linear regression.

If I understood correctly, we are coming from the following equation:

$$ y=w_0 x_0 + w_1 x_1 + ... + w_n x_n = \sum_{j=0}^{n} w^T_j x_j $$

We like to calculate the parameters $w$, so we can make predictions for new values of $x$.

To calculate so, we can "transform" the equation to a closed-form equation (calculate our searched variable $W$).

$$ W=(X^{⊤}X)^{−1}X^{⊤}y $$

So far, I understand the meaning of it and the power. What I don't understand exactly is the "transformation" part.

So far I know:

  • $X^TX$ : gives me a matrix with same cols and rows (square matrix).
  • $A^{-1}$ : inverse matrix ( $= \frac{1}{det(A)}A$). To calculate the determinant, we need to have a square matrix and it needs to be regular.

Why we calculate $X^TX$ and take the inverse to calculate it further? Does anyone have some good literature tips (maybe general) for solving math problems?

Our slides in the university looks as follow (even where they start is confusing for me):

lecture slide from ML course

  • 1
    $\begingroup$ regretfully, i will confess that I do things the other way around: when I see an expression like $(X^\top X)^{-1} X^\top y$ somewhere in linear algebra, I think to myself: "Oh, good, this behaves like a least squares solution"! We cannot get direct insight into the regression coefficients by considering individual elements of $X,y$. But maybe we can say this much at an intuitive level: $X^\top y$ calculates something similar to the coefficients if we were fitting independent regressions, and $(X^\top X)^{-1}$ corrects for the effects of the correlation between the X's. $\endgroup$ Commented Mar 31 at 17:49
  • $\begingroup$ Thx a lot for this fast answer. "Correction for the effects of the correlation between the X's " makes sense. I just wouldn't be able to come to this formula naturally. $\endgroup$
    – Pascal A.
    Commented Apr 1 at 8:12
  • $\begingroup$ Ah, if you're looking for a derivation of the formula, that can be done if you know how to do calculus with vectors. The formula pops out of the fact that $\mathbf{w}$ minimizes $\Vert\mathbf{y} - \mathbf{X}\mathbf{w}\Vert_2^2$. $\endgroup$ Commented Apr 1 at 16:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.