20
$\begingroup$

An embarrassingly simple question -- but it seems it has not been asked on Cross Validated before:

  1. What is the definition of a regression model?

Also a support question,

  1. What is not a regression model?

With regards to the latter, I am interested in tricky examples where the answer is not immediately obvious, e.g. ARIMA or GARCH.

$\endgroup$
0

5 Answers 5

15
$\begingroup$

I would say that "regression model" is a kind of meta-concept, in the sense that you will not find a definition of "regression model", but more concrete concepts such as "linear regression", "non-linear regression", "robust regression" and so on. This in the same way as in mathemathics we usually do not define "number", but "natural number", "integers", "real number", "p-adic number" and so on, and if somebody will want to include the quaternions among numbers so be it! it doesn't really matter, what matters is what definitions is used by the book/paper you are reading at the moment.

Definitions are tools, and essentialism, that is discussing what is the essence of ..., what a word really means, are seldom worthwhile.

So, what distinguishes a "regression model" from other kinds of statistical models? Mostly, that there is a response variable, which you want to model as influenced by (or determined by) some set of predictor variables. We are not interested in influence the other direction, and we are not interested in relationships among the predictor variables. Mostly, we take the predictor variables as given, and treat them as constants in the model, not as random variables.

The relationship mentioned above can be linear or nonlinear, specified in a parametric or nonparametric way, and so on.

To delineate from other models we better have a look at some other words often taken to denote something different for "regression models", like "errors in variables", when we accept the possibility of measurement errors in the predictor variables. That could well be included in my description of "regression model" above, but is often taken as an alternative model.

Also, what is meant might vary among fields, see What is the difference between conditioning on regressors vs. treating them as fixed?

To repeat: what matters is the definition used by the authors you are reading now, and not some metaphysics about what it "really is".

$\endgroup$
5
  • 1
    $\begingroup$ I agree with the essence of your answer. My question was motivated by having encountered statements about regression models that left me wondering what the statement really applies to (and what it does not apply to). Of course, now you could say, "use your best judgement and check the details carefully", but sometimes I might wish to reject the hypothesized statement right away saying that it is not true in general (perhaps true only in a very specific case). Then I need a definition to refer to. There are of course more such situations where having a precise definition is useful. $\endgroup$ Mar 16, 2016 at 13:12
  • 1
    $\begingroup$ Then you shouls ask specific questions about those uses you have encountered, with references. $\endgroup$ Mar 16, 2016 at 13:18
  • 2
    $\begingroup$ I don't intend to be picky, but think about it: someone asks you what you are doing, you say "I am analyzing/forecasting/testing [something] using regression models." -- "What is a regression model?" -- (Silence). Or a situation in an introductory econometrics class: "Professor, what is a regression model?" -- (No answer). I think these are very natural questions, so it would be nice to have an answer. $\endgroup$ Mar 16, 2016 at 19:24
  • 2
    $\begingroup$ Yes, it would be nice to have an answer, but I am not sure there is one canonical answer all can agree about. I got a very different idea of regression from a statistical book such as Seber: "Linear Regression Analysis" as from a text in econometrics. But some ideas all can agree about. I guess it is really a family of models. Then we can ask what is the common core of all this models. $\endgroup$ Mar 16, 2016 at 20:44
  • 1
    $\begingroup$ Perhaps you will be interested in a related question of mine: Definition of a simple linear regression model. $\endgroup$ Feb 13, 2020 at 7:28
13
$\begingroup$

Two nice answers were already given, but I'd like to add my two cents.

In regression case we have some random variables $Y$ and $X_1,\dots,X_k$. The variables have some unknown distribution and complicated covariance structure. We simplify this problem to focusing solely on conditional distribution, or more precisely on conditional expectation of $Y$ given the other variables. We simplify it to

$$ \mu = E(y|x_1,\dots,x_k) = f(x_1,\dots,x_k) $$

Where $f$ is a function of predictors that can take different forms (linear, non-linear) depending on particular regression model and $\mu$ is a mean of some distribution when thinking of regression models in terms of generalized linear models. In GLM's $\mu$ can be location of Poisson, Binomial, Gamma etc. distributions. With $L_1$ regularized regression it is a location of Laplace distribution, for robust model minimizing Huber loss so called Huber density is used. In case of quartile regression we focus on other feature of distribution, we estimate $\mu$ that is distribution's quartile rather then expected value.

So instead of looking on full joint distribution, we focus on conditional distribution of $Y$. This simplification is a key feature of regression models.

$\endgroup$
3
  • $\begingroup$ Thanks. Intuition doesn't hurt, although I am looking for a more formal definition that I could throw at someone who asked me, So what is a regression model anyway? and then tried to pick on details. $\endgroup$ May 6, 2016 at 16:47
  • $\begingroup$ @RichardHardy I think that this is the key feature of regression models that is shared by all of them. $\endgroup$
    – Tim
    May 6, 2016 at 16:49
  • 3
    $\begingroup$ I think this answer is a correct and useful approach, but it needs to be generalized so it can apply to situations commonly thought of as "regression" (including GLMs, multiplicative errors, regression with transformed responses, quantile regression, and so on). More broadly, a regression model specifies one or more properties of the entire distribution of the response $y$ in terms of the values of the regressors (within specific ranges, random or fixed). In particular, it can go far beyond merely specifying the expectation or assuming an additive error. $\endgroup$
    – whuber
    May 6, 2016 at 18:54
2
$\begingroup$

Some thoughts based on the literature:

F. Hayashi in Chapter 1 of his classic graduate textbook "Econometrics" (2000) states that the following assumptions comprise the classical linear regression model:

  1. Linearity
  2. Strict exogeneity
  3. No multicollinearity
  4. Spherical error variance
  5. "Fixed" regressors

Wooldridge in Chapter 2 of his classic introductory econometrics textbook "Introductory Econometrics: A Modern Approach" (2012) states that the following equation defines the simple linear regression model:

$$y=\beta_0+\beta_1 x+u.$$

Greene in Chapter 2 of his popular econometrics textbook "Econometric Analysis" (2011) states

The classical linear regression model consists of a set of assumptions about how a data set will be produced by an underlying “data-generating process.”

and subsequently gives a list of assumptions similar to that of Hayashi's.

Regarding the OP's interest in the GARCH model, Bollerslev "Generalized autoregressive conditional heterosedasticity" (1986) includes a phrase "the GARCH regression model" in the title of section 5 and also in the first sentence of that section. So the father of the GARCH model did not mind calling GARCH a regression model.

$\endgroup$
4
  • 1
    $\begingroup$ Your three references are all restricted to the linear regression model, but your question is wider than that. (Thus, using this as argument in your answer to another post, which I presume spawned the interest in this issue, is I think not completely valid.) If you'd say that latent variable models are not regression models, then using the immediate connection with measurement errors, regression models with measurement errors would no longer be regression models. Seems odd to me. Wiki just says that a reg model relates indep vars to dep in the sense that $Y\approx f(X, \beta)$. $\endgroup$
    – hejseb
    Sep 23, 2015 at 18:23
  • $\begingroup$ True, my examples are for linear regression models; that is what I was able to find in reliable sources such as these textbooks that are widely used and have become classic. I do not trust Wikipedia that much for statistical and econometric questions. Anyway, even in Wikipedia there is a chapter "Underlying assumptions" that is similar to what I have cited from the textbooks. Regarding the other post, could you post the relevant part of your comment there so that I could respond there? In this post I did not say anything about latent variable models, but it's good to hear you opinion. $\endgroup$ Sep 23, 2015 at 18:35
  • 3
    $\begingroup$ Why point 3, "no multicollinearity"? I have never seen that used as an assumption in the proof of some result! $\endgroup$ Mar 16, 2016 at 13:00
  • 1
    $\begingroup$ @kjetilbhalvorsen, please do not hold me responsible over what is written in a textbook which I am not the author of. But thanks for the comment, of course, and even more for the answer! $\endgroup$ Mar 16, 2016 at 13:07
2
$\begingroup$

Definition and delimitation of regression model

In the past me too shared your perplexity about this point. You refers on econometrics literature and me too refers primarily on that. Unfortunately most econometrics books do not helped much. However I achieved a clearer view that seems me consistent.

What is the definition of a regression model?

It seems me that the correct definition of Regression is as synonym for Conditional Expectation Function (CEF); this definition come from statistical literature. Therefore we can realize that all depends from the joint distribution of the random variables involved $D(y,X)$. The regression is

$E[y|X]=g(X)$

eventually we can speak of regression in error form representation

$y=E[y|X]+\epsilon$

read here for more about that: Regression and the CEF

Frequently people speak about linear regression (model), indeed this is the king object of econometrics. But from previous definition we can realize that linear regression is an explicit specification for $g(X)$. Moreover is possible to show that the true meaning of the famous mean independence assumption $E[\epsilon|X]=0$ is a restriction on $D(y,X)$; it imply linearity for the CEF.

Already here we can realize that assume both linearity for the regression and mean independence for his error is redundant. Worse, most econometrics books speak about Exogeneity assumption in place of mean independence assumption and attribute to it a crucial but strongly different meaning; meaning that cannot be attributed to a regression. Worse again, sometimes this assumption is given in the form $E[\epsilon X]=0$ that is always true by construction in all regressions! This fact reveal undoubtedly the contradiction. Two book the you (Richard Hardy) cited in your reply, Wooldridge 2012 and Greene (2011), are among them.

The core of the problem swing around a conflation among statistical and causal concepts. Indeed exogeneity is, or should be, a causal concept and not an assumption about regression.

What is not a regression model?

A structural equation is not a regression equation (model). The conflation between the two concepts seems me the root of the problems in econometrics literature. Indeed when econometric authors speak about Exogeneity (in any form defined) have in mind something like a structural equation not a regression equation. These my replies go deep about those point:

How would econometricians answer the objections and recommendations raised by Chen and Pearl (2013)?

Under which assumptions a regression can be interpreted causally?

What is the relationship between minimizing prediction error versus parameter estimation error?

moreover other conflation is among the meaning of regression intended as theoretical quantity and properties of his estimator, primarily OLS estimator. For example the “No multicollinearity” assumption is frequently intended as a necessity for uniqueness of OLS estimator, it is an algebraic condition that deal with the data at hand and have few to share with the statistical properties of the random variables involved.

Finally, all above for to said (also) that I do not suggest the use of word "regression" as a meta-concept in the sense suggested by kjetil b halvorsen; Indeed I fear that conflation between regression and structural equation come from that. Indeed concept like conditional quantile function can be interesting but refers on that as quantile regression is (widespread) bad custom. Moreover model like GARCH have much to share with the concept of Skedastic function different from regression one. About model like ARIMA I said that: AR subcase are surely regressions; ARMA is a regression that include unobservable terms; ARIMA looks like a regression but the use of integrated series can bring ad hoc statistical issues.

$\endgroup$
2
  • $\begingroup$ That was helpful, thank you! In this question I probably intended (a while ago, thus only probably) to put causality and structural causal modeling aside and focus on the statistical/probabilistic definition. Examples such as quantile regression and various forms of GARCH were of interest. By your answer, quantile regression does not deserve the name of a regression model as it does not focus on CEF. Nor does GARCH as it focuses on the conditional variance function, leaving the CEF be anything we find appropriate. $\endgroup$ Dec 27, 2021 at 12:25
  • $\begingroup$ your comment is fair, it suggest me to add something. $\endgroup$
    – markowitz
    Dec 27, 2021 at 13:52
0
$\begingroup$

You can split the question into: (i) "What is a model? and (ii)"What is regression?"

A "model" is any given way of making a prediction, in the sense of "all models are wrong, but some are useful".

"Regression" is the process of adjusting a model with the intention that its predictions become useful.

Typically some class of models is chosen, that is expected (due to some prior knowledge of the problem) to be capable of producing usefully accurate predictions (better than random, and better other models of similar cost and complexity).

Then certain parameters of the model are adjusted to improve its performance, by some measure of performance that is deemed appropriate for the problem the user wishes the model to solve.

See: https://en.wikipedia.org/wiki/Regression_analysis and https://www.investopedia.com/terms/r/regression.asp

Consequently, "What is not a regression model", would be where either or both (i) no prediction is made, (ii) no adjustment is made to improve the prediction. Note that even "descriptive" models still constrain the expected distribution of observations of the thing that they model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.