12
$\begingroup$

I used to say that OLS is an estimation technique and should never be confused with the type of model on which it is applied. Thus a phrase like "I have an OLS model" would not make sense to me, strictly speaking. (I would usually be able to guess what people mean, though.) However, in a comment under this post Dave offers a point to the contrary (if I am interpreting it correctly):

I’m not actually sold on OLS referring to an estimation technique rather than a model. Yes, we can estimate the coefficients of a linear model many ways and can apply minimization of square loss to estimating the coefficients of a nonlinear regression. However, the “ordinary” in OLS suggests to me an interest in a linear model whose coefficients are estimated by minimizing square loss.

Wikipedia's article on OLS seems to contain a similar message:

In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model (with fixed level-one effects of a linear function of a set of explanatory variables) by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable (values of the variable being observed) in the input dataset and the output of the (linear) function of the independent variable.

(emphasis is mine)

So strictly speaking, does the use of OLS imply the model is linear in parameters? In other words, does the term OLS refer to an estimation technique and linearity of the model at once?

I would like the answers to focus on the theoretical aspects of the issue. The fact that a lot of people misuse statistical terms such as OLS is of less concern for me.

$\endgroup$
15
  • $\begingroup$ Looking at the least-squares tag description, we only see reference to an estimation technique, not a model. However, the tag is LS, not OLS. On the other hand, a large part of the tag description actually applies to OLS but not necessarily to other versions of LS – without stating that explicitly. $\endgroup$ Dec 8, 2022 at 14:51
  • $\begingroup$ Do you want to add a references tag? I imagine an acceptable answer would refer to some commonly accepted gold standard reference. I wonder if there is an accepted explanation of what the "ordinary" stands for in OLS, and what "non-ordinary" least squares would be. $\endgroup$ Dec 8, 2022 at 15:07
  • 1
    $\begingroup$ @StephanKolassa, the tag will not hurt; I have added it. Thanks. Regarding "non-ordinary", I suppose nonlinear LS is a good example. It could be used on a model that is not linear in parameters. $\endgroup$ Dec 8, 2022 at 15:08
  • $\begingroup$ My two cents are that the answer to your question should still be no. Whenever you are given $(y,X)$ you can run OLS on that, whether the model you have in mind/that nature used to generate $(y,X)$ is linear in parameters or not (say, because nature used probit or logit). $\endgroup$ Dec 8, 2022 at 15:37
  • $\begingroup$ @ChristophHanck, would that not confuse the data generating process (DGP) with the model? Regardless of the DGP behind a sample of data, I may be fitting a model using OLS. Does that necessarily imply the model is linear in parameters? $\endgroup$ Dec 8, 2022 at 15:41

2 Answers 2

11
$\begingroup$

Ordinary least squares regression is a special case of least squares regression.

With least squares regression we try to find a fit $\hat{y}_i({\bf{x}}_i,\boldsymbol{\beta})$ to datapoints $y_i$ by minimising the sum of (weighted) squared residuals.

$$\text{given data $\bf{x}_i$ and $y_i$, and weights $w_i$ find $\boldsymbol{\beta}$ that minimises:} \quad L = \sum_{i = 1}^n w_i [y_i-\hat{y}_i({\bf{x}}_i,\boldsymbol{\beta})]^2$$

OLS is the special case when the weights are equal $w_i = 1$ and the model is a linear combination

$$\hat{y}_i({\bf{x}}_i,\boldsymbol{\beta}) = \beta_1 f_1({\bf x}) + \beta_2 f_2({\bf x}) + \dots +\beta_p f_p({\bf x}). $$

OLS is by definition using a linear model.


But not all methods that use linear models are OLS. For instance think of GLM, quantile regression, lasso/ridge or Bayesian modelling, which can use a linear model but with a different cost function.

$\endgroup$
10
  • 1
    $\begingroup$ Thank you! I suppose that answers it. I will wait for any other views for a bit. Regarding "not all linear models are OLS", I have a hard time with the phrase. A model cannot be OLS, since OLS is an estimation technique (plus probably a model description). $\endgroup$ Dec 8, 2022 at 15:07
  • 1
    $\begingroup$ Not to be the party pooper, but would a nonlinear model fitted to minimize the sum of squares also be "least squares", and if the observations were weighted, "weighted least squares", and thus if all the weights are equal... "ordinary least squares"? Or the other way around, if such a model is not "least squares", then what would it be? $\endgroup$ Dec 8, 2022 at 15:12
  • $\begingroup$ @StephanKolassa would a nonlinear model fitted to minimize the sum of squares also be "least squares" yes that would be least squares, not? Sometimes written as non-linear least squares to be extra clear. $\endgroup$ Dec 8, 2022 at 15:14
  • 2
    $\begingroup$ Btw. I agree with critism of the phrase "I ran an OLS model"; Strictly speaking you don't run a model. $\endgroup$ Dec 8, 2022 at 15:20
  • 1
    $\begingroup$ Just for my own sanity about definitions: OLS is simply an estimation method (estimating distance from a prediction line and raw values, hence the "squares" they resemble), whereas a model is something that is theoretical ("I think x influences y")? I think people on this site are a lot more laser-focused on these things than I am used to. $\endgroup$ Dec 8, 2022 at 15:37
0
$\begingroup$

By means of OLS one can estimate non linear relations provided they are purely additive or purely multiplicative (log-additive). For instance, quadratic relations like this: $$ y_t = a + bx_t + cx^2_t + e_t, $$ are perfectly fitted within the OLS framework and of course, their variants and extenions; but not like this: $$ y_t = a\log b^{2x_{1,t}}+ cx_{2,t}^{b}e_t. $$

The latter can arise in structural specifications where typically the parameters a,b,c is what you can get from data, so that reduced version models are simply ill designed to make good forecasts.

$\endgroup$
7
  • 2
    $\begingroup$ The post is explicitly about linearity in the parameters, so I do not see how your answer helps here. $\endgroup$
    – Michael M
    Dec 9, 2022 at 22:54
  • 2
    $\begingroup$ Welcome to Cross Validated! This is very often a good point to make, but as @MichaelM says, using "linearity in the parameters" shows the question asker already understands OLS can be used to fit your 1st model (the response $y$ is a linear function of the parameters $a$, $b$, & $c$) but not your 2nd model (the response $y$ is a non-linear function of the parameters $a$, $b$, & $c$). (My favourite example, by the way, is that you can fit a sine wave of known period using OLS.) $\endgroup$ Dec 9, 2022 at 23:50
  • $\begingroup$ Thanks for the point. If you want me to delete the note, tell me. But obviously, if the structural model is $$ y_t = b^2x_t $$ and you estimate $y_t = \beta x_t,$ you can obtain two solutions or none from the structural version. So the question always boils down to the relation of deep theory and data. $\endgroup$
    – Samuel
    Dec 10, 2022 at 23:37
  • $\begingroup$ Your second model is too ambiguous to serve as an example or counterexample. What is the argument of $\log$? Is the "$b$" in $x_{2,t}^b$ a power or an index? $\endgroup$
    – whuber
    Dec 11, 2022 at 0:11
  • $\begingroup$ I guess my initial point was a mix of a multiplicative and additive model, but there are innumerable alternatives. Thank you (corrected). $\endgroup$
    – Samuel
    Dec 12, 2022 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.