20
$\begingroup$

I saw a thread about this but they seem to have got caught up and dive into statistical theory, trying to explain different things than this concept. So can anyone explain the difference between these two regressions in a simple way?

$\endgroup$
2
  • 6
    $\begingroup$ Strictly, least squares is a method of estimation and linear regression refers to fitting a model that is linear in the parameters. Historically, regression is about summarizing the mean response as a function of predictors, but other flavours of regression extend that (or contradict it if you will, so quantile regression is not about means, but can be linear). Talk of OLS regression is in my experience most common among economists and defensible as in-field short-hand (and also objectionable as mixing different ideas together). (Detail: least squares is more than OLS, ordinary least squares.) $\endgroup$ – Nick Cox May 11 at 12:55
  • 2
    $\begingroup$ No. There's non-linear least squares; like fitting a sinusoidal wave to noisy periodic data or Michaelis Menten kinetics equations. There's also weighted linear regression, which doesn't actually minimize the squared error. $\endgroup$ – AdamO May 11 at 16:00
17
$\begingroup$

An explanation rather depends on what your background is.

Suppose you have some so-called independent variables $x_1,x_2,\ldots, x_k$ (they do not have to be independent of each other) where each $x_i$ takes takes values $x_{i,1}, x_{i,2}\ldots, x_{i,n}$ and you want a regression for a dependent variable $y$ taking values $y_{1}, y_{2}\ldots, y_{n}$. Then you are trying to find a function $f(x_{1,j}, x_{2,j},\ldots, x_{k,j})$ of the independent variables which in some sense minimises the loss from using that function as some measure across the observations comparing all the $y_j$ and their corresponding $f(x_{1,j}, x_{2,j},\ldots, x_{k,j})$

  • Linear regression restricts the possible $f$ to those of the form $f(x_{1,j}, x_{2,j},\ldots, x_{k,j})=\beta_0+\beta_1x_{1,j}+\beta_2x_{2,j}+\ldots+\beta_kx_{k,j}$ for real values $\beta_0,\beta_1,\beta_2, \ldots ,\beta_k$.

  • Least squares regression uses a loss function of the form $\sum\limits_{j=1}^n (y_j - f(x_{1,j}, x_{2,j},\ldots, x_{k,j}))^2$ which you want to minimise by choosing a suitable $f$.

Ordinary Least Squares Linear Regression combines the linear form of estimator and minimising the sum of the squares of the differences, so both requirements. But other forms of regression may only use one or even neither of them. For example, logistic regression can be seen as not being linear (it is not least-squares either, instead using maximum likelihood techniques), while robust regression typically is not a simple least squares calculation though may be linear

$\endgroup$
5
  • $\begingroup$ Thank you @Henry this is the most simple answer. Also thanks to nick cox and graham wright for making it easy to understand this concept. I've been doing statistical analysis for a long time but the reason I asked this question was that people misuse these terms and I wanted to make sure of the correct versions of using them. $\endgroup$ – Atilla Colak May 11 at 13:28
  • 1
    $\begingroup$ "logistic regression can be seen as not being linear" - this is potentially a bit misleading, as in the context you're writing, it sounds like you're giving an example of something that is one of the two but not the other. But logistic regression isn't estimated by least squares either! So might be good to slightly reword this... $\endgroup$ – Silverfish May 12 at 12:45
  • $\begingroup$ @Silverfish You are correct but my intention on the logistic regression was just an example that readers may have come across, where it is not really about fitting a straight line (or hyperplane) to a set of points involving the dependent and independent variables, without addressing the question of whether it was a least-squares regression $\endgroup$ – Henry May 12 at 13:06
  • $\begingroup$ Yes, I understood what you meant - I'm just thinking that if I hadn't know that fact, and was reading your answer, it (accidentally and implicitly) suggests that logistic regression might actually be least-squares. "X and Y are not the same: foo is X but not Y, and bar is Y but not X" is just such a common rhetorical device! A minor tweak of the wording might leave a clearer impression of what you actually mean. $\endgroup$ – Silverfish May 12 at 13:25
  • 1
    $\begingroup$ Yeah, I agree with @Silverfish. This is the best answer in my opinion. I love how you set up the definitions, and then distinguished the two terms from the question. However, logistic regression is linear, but has a different loss function. Consider that OLSR and logistic regression are both examples of generalized linear models. Both are linear: the predicted natural parameters are a linear function of the independent variables. They differ in loss function. I think the example should be in its own paragraph and clarified. $\endgroup$ – Neil G May 21 at 22:24
17
$\begingroup$

Least squares is the processes of minimizing the sum of squared errors from some model. Given a function $f$ which depends on parameters $\theta$, the least squares estimates of $\theta$ are

$$ \hat{\theta} = \underset{\theta \in \mathbb{R}^p}{\mbox{argmin}} \left\{ \sum_i (y_i - f(x_i ; \theta))^2 \right\}$$

If you look at the optimization for linear regression, it looks a lot like this.

$$ \hat{\beta} = \underset{\beta \in \mathbb{R}^p}{\mbox{argmin}} \left\{ \sum_i (y_i - x_i^T \beta)^2 \right\}$$

An important difference being that the function in linear regression is linear in its parameters, whereas $f$ is not necessarily so. It is sensible to say that linear regression is fit via least squares.

However, some things may apply to linear regression which may not apply to all functions fit via least squares. The assumptions of linear regression (normality of residuals, independence, homogeneity of variance, and getting the functional form right) permit inference via confidence intervals and hypothesis tests. If the data used to fit your model via least squares do not satisfy those assumptions, the inferences may not have the properties we wish them to have.

If ever there was a time to say it, it is now; linear regression and least squares are same same but different.

$\endgroup$
0
-2
$\begingroup$

Both "Linear Regression" and "Ordinary Least Squares" (OLS) regression are often used to refer to the same kind of statistical model, but for different reasons. We call the model "linear" because it assumes that the relationship between the independent and dependent variables can be described by a straight line. We call it "least squares" because we estimate the parameters of the model by finding the parameters that minimize the squared error terms ("least squares"). Technically we could use other, more complicated methods (like maximum likelihood estimation) to estimate a linear model, and sometimes we need to do this (like when you have a multilevel linear model) because there are other complexities in the model that make OLS estimation not possible. In those cases we have a linear regression model that is not estimated via least squares. On the other hand the least squares method really only works for linear models. So in practice we often treat "OLS" and "linear model" as meaning same thing, even though one refers to the assumptions of the model and the other refers to the way it is estimated.

$\endgroup$
3
  • 2
    $\begingroup$ It isn't quite true that the relationship has to be a straight line: youtube.com/watch?v=rVviNyIR-fI. $\endgroup$ – Dave May 11 at 12:23
  • $\begingroup$ Sure, You linear models are robust to violations of linearity and can use polynomial terms and what not and to model nonlinear processes. But the word "linear" is in the name of the model because there is an assumption of linearity in there somewhere and I wanted to keep things simple to focus on the specific question asked in the OP, who sounds like they are just starting out in stats and might prefer a less technical explanation. $\endgroup$ – Graham Wright May 11 at 12:39
  • 4
    $\begingroup$ The intent is admirable, but this runs some risk of over-simplification for the reason @Dave gave, and others, principally that using least squares for nonlinear models is an art form in its own right. $\endgroup$ – Nick Cox May 11 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.