1
$\begingroup$

The best line fit can be found analytically by the least squares method. So can we say that linear regression (least squares) has an optimizer?

For example, for logistic regression I can use an optimizer, gradient descent. But for linear regression, where the best model parameters can be found analytically, does it make sense to say that I use an optimizer? An optimizer in general can be any algorithm that takes as input a model, model parameters, evaluator (function that tells you how good particular parameters are), and produces as output the best model parameters.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Well you can always fit linear regression using gradient descent, which can be computationally beneficial, e.g. when design matrix $X$ is very large $\endgroup$
    – Łukasz Grad
    May 21, 2017 at 16:49
  • $\begingroup$ Your question actually hinges on your definition of "best". Without defining it, you can't even tell if you're optimizing by that definition. Also beware conflating the optimization method with the criterion you're optimizing; the two are distinct. $\endgroup$
    – Glen_b
    May 21, 2017 at 17:02

1 Answer 1

Reset to default
0
$\begingroup$

We had many related discussions in CV, for example, My answer here.

When we solve a least square problem, no matter which algorithm to use, we are doing an optimization: $\text{minimize}~ \|Ax-b\|^2$.

The answer to your question is really depending on how to define "optimizer". And even using analytical solution, there are also algorithms needed, for example QR decomposition, details can be found here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.