12
$\begingroup$

What is the difference between least squares and linear regression? Is it the same thing?

$\endgroup$
  • 4
    $\begingroup$ I'd say that ordinary least squares is one estimation method within the broader category of linear regression. It's possible though that some author is using "least squares" and "linear regression" as if they were interchangeable. $\endgroup$ – Matthew Gunn Feb 2 '17 at 6:55
  • 1
    $\begingroup$ If you're doing ordinary least squares, I'd use that term. It's less ambiguous. $\endgroup$ – Matthew Gunn Feb 2 '17 at 7:03
  • 1
    $\begingroup$ See also what is a regression model. $\endgroup$ – Richard Hardy Feb 2 '17 at 8:22
22
$\begingroup$

Linear regression assumes a linear relationship between the independent and dependent variable. It doesn't tell you how the model is fitted. Least square fitting is simply one of the possibilities. Other methods for training a linear model is in the comment.

Non-linear least squares is common (https://en.wikipedia.org/wiki/Non-linear_least_squares). For example, the popular Levenberg–Marquardt algorithm solves something like:

$$\hat\beta=\mathop{\textrm{argmin}}_\beta S(\beta)\equiv \mathop{\textrm{argmin}}_\beta\sum_{i=1}^{m}\left[ y_i-f(x_i,\beta) \right]^2$$

It is a least squares optimization but the model is not linear.

They are not the same thing.

$\endgroup$
13
$\begingroup$

In addition to the correct answer of @Student T, I want to emphasize that least squares is a potential loss function for an optimization problem, whereas linear regression is an optimization problem.

Given a certain dataset, linear regression is used to find the best possible linear function, which is explaining the connection between the variables.

In this case the "best" possible is determined by a loss function, comparing the predicted values of a linear function with the actual values in the dataset. Least Squares is a possible loss function.

The wikipedia article of least-squares also shows pictures on the right side which show using least squares for other problems than linear regression such as:

  • conic-fitting
  • fitting quadratic function

The following gif from the wikipedia article shows several different polynomial functions fitted to a dataset using least squares. Only one of them is linear (polynom of 1). This is taken from the german wikipedia article to the topic.

enter image description here

$\endgroup$
  • 2
    $\begingroup$ We can argue the non-linear examples in the animation are actually still linear in the parameters. $\endgroup$ – Firebug Feb 2 '17 at 12:21
  • 2
    $\begingroup$ True, yet the model relation between the target and the input variable is non linear. Would you yet call the fitting "linear regression"? I would not. $\endgroup$ – Nikolas Rieble Feb 2 '17 at 14:32
  • 4
    $\begingroup$ We should distinguish between "linear least squares" and "linear regression", as the adjective "linear" in the two are referring to different things. The former refers to a fit that is linear in the parameters, and the latter refers to fitting to a model that is a linear function of the independent variable(s). $\endgroup$ – J. M. is not a statistician Feb 2 '17 at 19:52
  • 3
    $\begingroup$ @J.M. Many sources maintain that "linear" in "linear" regression means "linear in the parameters" rather "linear in the IVs". The WIkipedia article on linear regression is an example is an example. Here's another and another. Many statistics texts do the same; I'd argue it's a convention. $\endgroup$ – Glen_b Feb 6 '17 at 0:04
  • 1
    $\begingroup$ @Glen, prolly a later development than the stuff I read (I'm an old hand at this); they limited "linear regression" to fitting the model $y=mx+b$. $\endgroup$ – J. M. is not a statistician Mar 18 '17 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.