13
$\begingroup$

An embarrassingly simple question -- but it seems it has not been asked on Cross Validated before:

  1. What is the definition of a regression model?

Also a support question,

  1. What is not a regression model?

With regards to the latter, I am interested in tricky examples where the answer is not immediately obvious. For example,

  • What about latent variable models (e.g. ARIMA or GARCH)?
$\endgroup$
9
$\begingroup$

I would say that "regression model" is a kind of meta-concept, in the sense that you will not find a definition of "regression model", but more concrete concepts such as "linear regression", "non-linear regression", "robust regression" and so on. This in the same way as in mathemathics we usually do not define "number", but "natural number", "integers", "real number", "p-adic number" and so on, and if somebody will want to include the quaternions among numbers so be it! it doesn't really matter, what matters is what definitions is used by the book/paper you are reading at the moment.

Definitions are tools, and essentialism, that is discussing what is the essence of ..., what a word really means, are seldom worthwhile.

So, what distinguishes a "regression model" from other kinds of statistical models? Mostly, that there is a response variable, which you want to model as influenced by (or determined by) some set of predictor variables. We are not interested in influence the other direction, and we are not interested in relationships among the predictor variables. Mostly, we take the predictor variables as given, and treat them as constants in the model, not as random variables.

The relationship mentioned above can be linear or nonlinear, specified in a parametric or nonparametric way, and so on.

To delineate from other models we better have a look at some other words often taken to denote something different for "regression models", like "errors in variables", when we accept the possibility of measurement errors in the predictor variables. That could well be included in my description of "regression model" above, but is often taken as an alternative model.

Also, what is meant might vary among fields, see What is the difference between conditioning on regressors vs. treating them as fixed?

To repeat: what matters is the definition used by the authors you are reading now, and not some metaphysics about what it "really is".

$\endgroup$
  • 1
    $\begingroup$ I agree with the essence of your answer. My question was motivated by having encountered statements about regression models that left me wondering what the statement really applies to (and what it does not apply to). Of course, now you could say, "use your best judgement and check the details carefully", but sometimes I might wish to reject the hypothesized statement right away saying that it is not true in general (perhaps true only in a very specific case). Then I need a definition to refer to. There are of course more such situations where having a precise definition is useful. $\endgroup$ – Richard Hardy Mar 16 '16 at 13:12
  • $\begingroup$ Then you shouls ask specific questions about those uses you have encountered, with references. $\endgroup$ – kjetil b halvorsen Mar 16 '16 at 13:18
  • 2
    $\begingroup$ I don't intend to be picky, but think about it: someone asks you what you are doing, you say "I am analyzing/forecasting/testing [something] using regression models." -- "What is a regression model?" -- (Silence). Or a situation in an introductory econometrics class: "Professor, what is a regression model?" -- (No answer). I think these are very natural questions, so it would be nice to have an answer. $\endgroup$ – Richard Hardy Mar 16 '16 at 19:24
  • 1
    $\begingroup$ Yes, it would be nice to have an answer, but I am not sure there is one canonical answer all can agree about. I got a very different idea of regression from a statistical book such as Seber: "Linear Regression Analysis" as from a text in econometrics. But some ideas all can agree about. I guess it is really a family of models. Then we can ask what is the common core of all this models. $\endgroup$ – kjetil b halvorsen Mar 16 '16 at 20:44
7
$\begingroup$

Two nice answers were already given, but I'd like to add my two cents.

In regression case we have some random variables $Y$ and $X_1,\dots,X_k$. The variables have some unknown distribution and complicated covariance structure. We simplify this problem to focusing solely on conditional distribution, or more precisely on conditional expectation of $Y$ given the other variables. We simplify it to

$$ \mu = E(y|x_1,\dots,x_k) = f(x_1,\dots,x_k) $$

Where $f$ is a function of predictors that can take different forms (linear, non-linear) depending on particular regression model and $\mu$ is a mean of some distribution when thinking of regression models in terms of generalized linear models. In GLM's $\mu$ can be location of Poisson, Binomial, Gamma etc. distributions. With $L_1$ regularized regression it is a location of Laplace distribution, for robust model minimizing Huber loss so called Huber density is used. In case of quartile regression we focus on other feature of distribution, we estimate $\mu$ that is distribution's quartile rather then expected value.

So instead of looking on full joint distribution, we focus on conditional distribution of $Y$. This simplification is a key feature of regression models.

$\endgroup$
  • $\begingroup$ Thanks. Intuition doesn't hurt, although I am looking for a more formal definition that I could throw at someone who asked me, So what is a regression model anyway? and then tried to pick on details. $\endgroup$ – Richard Hardy May 6 '16 at 16:47
  • $\begingroup$ @RichardHardy I think that this is the key feature of regression models that is shared by all of them. $\endgroup$ – Tim May 6 '16 at 16:49
  • 3
    $\begingroup$ I think this answer is a correct and useful approach, but it needs to be generalized so it can apply to situations commonly thought of as "regression" (including GLMs, multiplicative errors, regression with transformed responses, quantile regression, and so on). More broadly, a regression model specifies one or more properties of the entire distribution of the response $y$ in terms of the values of the regressors (within specific ranges, random or fixed). In particular, it can go far beyond merely specifying the expectation or assuming an additive error. $\endgroup$ – whuber May 6 '16 at 18:54
2
$\begingroup$

Some thoughts based on the literature:

F. Hayashi in Chapter 1 of his classic graduate textbook "Econometrics" (2000) states that the following assumptions comprise the classical linear regression model:

  1. Linearity
  2. Strict exogeneity
  3. No multicollinearity
  4. Spherical error variance
  5. "Fixed" regressors

Wooldridge in Chapter 2 of his classic introductory econometrics textbook "Introductory Econometrics: A Modern Approach" (2012) states that the following equation defines the simple linear regression model:

$$y=\beta_0+\beta_1 x+u.$$

Greene in Chapter 2 of his popular econometrics textbook "Econometric Analysis" (2011) states

The classical linear regression model consists of a set of assumptions about how a data set will be produced by an underlying “data-generating process.”

and subsequently gives a list of assumptions similar to that of Hayashi's.

A curiosity relevant to the last bullet point of the OP: Bollerslev "Generalized autoregressive conditional heterosedasticity" (1986) includes a phrase "the GARCH regression model" in the title of section 5 and also in the first sentence of that section. So the father of the GARCH model did not mind calling GARCH a regression model.

$\endgroup$
  • 1
    $\begingroup$ Your three references are all restricted to the linear regression model, but your question is wider than that. (Thus, using this as argument in your answer to another post, which I presume spawned the interest in this issue, is I think not completely valid.) If you'd say that latent variable models are not regression models, then using the immediate connection with measurement errors, regression models with measurement errors would no longer be regression models. Seems odd to me. Wiki just says that a reg model relates indep vars to dep in the sense that $Y\approx f(X, \beta)$. $\endgroup$ – hejseb Sep 23 '15 at 18:23
  • $\begingroup$ True, my examples are for linear regression models; that is what I was able to find in reliable sources such as these textbooks that are widely used and have become classic. I do not trust Wikipedia that much for statistical and econometric questions. Anyway, even in Wikipedia there is a chapter "Underlying assumptions" that is similar to what I have cited from the textbooks. Regarding the other post, could you post the relevant part of your comment there so that I could respond there? In this post I did not say anything about latent variable models, but it's good to hear you opinion. $\endgroup$ – Richard Hardy Sep 23 '15 at 18:35
  • 3
    $\begingroup$ Why point 3, "no multicollinearity"? I have never seen that used as an assumption in the proof of some result! $\endgroup$ – kjetil b halvorsen Mar 16 '16 at 13:00
  • 1
    $\begingroup$ @kjetilbhalvorsen, please do not hold me responsible over what is written in a textbook which I am not the author of. But thanks for the comment, of course, and even more for the answer! $\endgroup$ – Richard Hardy Mar 16 '16 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.