A while ago I was trying (not entirely successfully) to figure out the definition of a regression model. Now I am narrowing it down to a simple linear regression and trying to identify (loosely speaking) where the basic model ends and the optional assumptions begin. Wooldridge "Introductory Econometrics: A Modern Approach" (6th edition, 2016) states the following at the beginning of Chapter 2:

In writing down a model that will “explain $y$ in terms of $x$,” we must confront three issues. First, since there is never an exact relationship between two variables, how do we allow for other factors to affect $y$? Second, what is the functional relationship between $y$ and $x$? And third, how can we be sure we are capturing a ceteris paribus relationship between $y$ and $x$ (if that is a desired goal)? We can resolve these ambiguities by writing down an equation relating $y$ to $x$. A simple equation is $$ y=\beta_0+\beta_1 x_1+u. \quad [2.1] $$ Equation $(2.1)$, which is assumed to hold in the population of interest, defines the simple linear regression model. (Emphasis is mine.)

This does not look complete enough to indicate anything about $y$ and $x$ probabilistically. As far as I understand, one could write such an equation for any variables $y$ and $x$, with any chosen constants $\beta_0$ and $\beta_1$, and it would hold as long as we choose the right values of $u$. So this does not look like a good candidate for a definition of a regression model to me.

Now on the other end of the spectrum we could have a linear model $$ y|x_1\sim i.i.D(\beta_0+\beta_1 x_1,\sigma^2) $$ for some density $D$ characterized by a location and a scale parameter. This is probably too restrictive as a definition of a simple linear regression [model], because I think we could still call it one if the scale above was a function of $x_1$ or if the distribution was something else than the specific $D$.

In between too loose and too restrictive, we can have models like $$ \mathbb{E}(y|x_1)=\beta_0+\beta_1 x_1, $$ or $$ \mathbb{E}(y|x_1)=\beta_0+\beta_1 x_1, \quad \text{Var}(y|x_1)=\sigma^2 $$ and other. Somewhere there I expect to find a definition that makes the most sense in the context of the term "a simple linear regression [model]".

So what is the definition of a simple linear regression [model]?

  • $\begingroup$ I began my answer at stats.stackexchange.com/a/148713/919 with a general formulation of a probabilistic regression model. The discussion that follows covers all your examples and more, so I suspect it may be a complete answer to your questions here. $\endgroup$ – whuber Feb 12 at 15:13
  • $\begingroup$ (2.1) looks pretty much like a simple linear regression model, though it would be usual to have $u$ have mean $0$ and be uncorrelated with the $x_i$. This allows finding in some sense optimal estimates of $\beta_0$ and $\beta_1$. A stronger assumption would be that $u$ is independent of the $x_i$ and i.i.d. (or even be normally distributed). $\endgroup$ – Henry Feb 12 at 15:26
  • $\begingroup$ @whuber, in your answer, A linear model of a linear relationship with additive errors as well as the other items in that list suffer from the same problem as $(2.1)$ above. Or is the problem I have indicated not really a problem? You do write A model has *additive errors when $f$ is linear in $\varepsilon$. In such cases it is always assumed that $\mathbb{E}(\varepsilon)=0$*. After this addition, is the linear regression model complete? If so, what does it tell us probabilistically about $y$ and $x$? $\endgroup$ – Richard Hardy Feb 12 at 15:28
  • $\begingroup$ I urge you to look at the initial formulation of the model as $Y = f(X,\theta,\epsilon).$ Everything else is just special cases of that presented in order to address the specific question of what "linear" might mean. This general formulation has none of the issues you attach to (2.1). $\endgroup$ – whuber Feb 12 at 15:30
  • 1
    $\begingroup$ I'm sorry? Given $x_1,$ the distribution of $y$ is the distribution of $u,$ shifted by $3+2x_1:$ you know everything about that conditional distribution that you know about $u.$ In some circumstances you might specify that $u$ has a Normal$(0,\sigma^2)$ distribution; it other circumstances you might only require that $u$ have zero mean and finite variance; but regardless, it is obvious you do have information about the conditional distribution of $y.$ $\endgroup$ – whuber Feb 12 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.