1
$\begingroup$

In many online machine learning courses and videos(such as Andrew Ng's coursera course), when it comes to regression (for example regressing $Y$ on features $X$), althouth we have the closed form estimator for regression coefficient $\widehat{\beta}=(X'X)^{-1}X'Y$, and based on this we could come out with the prediction at $X_i=x$ as $x'\widehat{\beta}$. This is simple and no numerical optimization is needed. My questios are:

  1. given the simplicity of the closed form regression estimator (and predictor), why do machine learning courses typically ignore it, and only focus on gradient descent?

  2. what's the merits of teaching regression in this way?

  3. Also, what's the relative merits of gradient descent in both practical performance?

Improve this question
$\endgroup$
3
  • $\begingroup$ arguably, this is not a duplicate insofar as the other question answers why we might prefer gradient descent vs direct solution from a practical, problem solving perspective, whereas this question is pedagogical in nature. One good reason to teach gradient descent for linear regression in an ML course is that this same procedure extends to other differentiable models like logistic reg or neural nets, whereas the direct method does not. $\endgroup$
    – John Madden
    Commented 23 hours ago
  • 1
    $\begingroup$ @JohnMadden I am with you that this is not quite a duplicate. I may post an answer later if no one beats me to it. $\endgroup$
    – Dave
    Commented 23 hours ago
  • $\begingroup$ The merit of gradient descent is it is applicable to a much wider class of models. Whereas the closed form OLS estimator is applicable to only one model. $\endgroup$
    – jarbet
    Commented 23 hours ago

0

Reset to default