# “Least Squares” and “Linear Regression”, are they synonyms?

What is the difference between least squares and linear regression? Is it the same thing?

• I'd say that ordinary least squares is one estimation method within the broader category of linear regression. It's possible though that some author is using "least squares" and "linear regression" as if they were interchangeable. – Matthew Gunn Feb 2 '17 at 6:55
• If you're doing ordinary least squares, I'd use that term. It's less ambiguous. – Matthew Gunn Feb 2 '17 at 7:03
• See also what is a regression model. – Richard Hardy Feb 2 '17 at 8:22

Linear regression assumes a linear relationship between the independent and dependent variable. It doesn't tell you how the model is fitted. Least square fitting is simply one of the possibilities. Other methods for training a linear model is in the comment.

Non-linear least squares is common (https://en.wikipedia.org/wiki/Non-linear_least_squares). For example, the popular Levenberg–Marquardt algorithm solves something like:

$$\hat\beta=\mathop{\textrm{argmin}}_\beta S(\beta)\equiv \mathop{\textrm{argmin}}_\beta\sum_{i=1}^{m}\left[ y_i-f(x_i,\beta) \right]^2$$

It is a least squares optimization but the model is not linear.

They are not the same thing.

In addition to the correct answer of @Student T, I want to emphasize that least squares is a potential loss function for an optimization problem, whereas linear regression is an optimization problem.

Given a certain dataset, linear regression is used to find the best possible linear function, which is explaining the connection between the variables.

In this case the "best" possible is determined by a loss function, comparing the predicted values of a linear function with the actual values in the dataset. Least Squares is a possible loss function.

The wikipedia article of least-squares also shows pictures on the right side which show using least squares for other problems than linear regression such as:

• conic-fitting
• @Glen, prolly a later development than the stuff I read (I'm an old hand at this); they limited "linear regression" to fitting the model $y=mx+b$. – J. M. is not a statistician Mar 18 '17 at 16:25