What is the difference between least squares and linear regression? Is it the same thing?
$\begingroup$
$\endgroup$
3
-
4$\begingroup$ I'd say that ordinary least squares is one estimation method within the broader category of linear regression. It's possible though that some author is using "least squares" and "linear regression" as if they were interchangeable. $\endgroup$– Matthew GunnFeb 2, 2017 at 6:55
-
1$\begingroup$ If you're doing ordinary least squares, I'd use that term. It's less ambiguous. $\endgroup$– Matthew GunnFeb 2, 2017 at 7:03
-
1$\begingroup$ See also what is a regression model. $\endgroup$– Richard HardyFeb 2, 2017 at 8:22
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Linear regression assumes a linear relationship between the independent and dependent variable. It doesn't tell you how the model is fitted. Least square fitting is simply one of the possibilities. Other methods for training a linear model is in the comment.
Non-linear least squares is common (https://en.wikipedia.org/wiki/Non-linear_least_squares). For example, the popular Levenberg–Marquardt algorithm solves something like:
$$\hat\beta=\mathop{\textrm{argmin}}_\beta S(\beta)\equiv \mathop{\textrm{argmin}}_\beta\sum_{i=1}^{m}\left[ y_i-f(x_i,\beta) \right]^2$$
It is a least squares optimization but the model is not linear.
They are not the same thing.
answered Feb 2, 2017 at 7:09
$\begingroup$
$\endgroup$
10
In addition to the correct answer of @Student T, I want to emphasize that least squares is a potential loss function for an optimization problem, whereas linear regression is an optimization problem.
Given a certain dataset, linear regression is used to find the best possible linear function, which is explaining the connection between the variables.
In this case the "best" possible is determined by a loss function, comparing the predicted values of a linear function with the actual values in the dataset. Least Squares is a possible loss function.
The wikipedia article of least-squares also shows pictures on the right side which show using least squares for other problems than linear regression such as:
- conic-fitting
- fitting quadratic function
The following gif from the wikipedia article shows several different polynomial functions fitted to a dataset using least squares. Only one of them is linear (polynom of 1). This is taken from the german wikipedia article to the topic.
-
3$\begingroup$ We can argue the non-linear examples in the animation are actually still linear in the parameters. $\endgroup$– FirebugFeb 2, 2017 at 12:21
-
2$\begingroup$ True, yet the model relation between the target and the input variable is non linear. Would you yet call the fitting "linear regression"? I would not. $\endgroup$ Feb 2, 2017 at 14:32
-
4$\begingroup$ We should distinguish between "linear least squares" and "linear regression", as the adjective "linear" in the two are referring to different things. The former refers to a fit that is linear in the parameters, and the latter refers to fitting to a model that is a linear function of the independent variable(s). $\endgroup$ Feb 2, 2017 at 19:52
-
3$\begingroup$ @J.M. Many sources maintain that "linear" in "linear" regression means "linear in the parameters" rather "linear in the IVs". The WIkipedia article on linear regression is an example is an example. Here's another and another. Many statistics texts do the same; I'd argue it's a convention. $\endgroup$– Glen_bFeb 6, 2017 at 0:04
-
1$\begingroup$ @Glen, prolly a later development than the stuff I read (I'm an old hand at this); they limited "linear regression" to fitting the model $y=mx+b$. $\endgroup$ Mar 18, 2017 at 16:25