Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
An embarrassingly simple question -- but it seems it has not been asked on Cross Validated before:
Also a support question,
With regards to the latter, I am interested in tricky examples where the answer is not immediately obvious, e.g. ARIMA or GARCH.
I would say that "regression model" is a kind of meta-concept, in the sense that you will not find a definition of "regression model", but more concrete concepts such as "linear regression", "non-linear regression", "robust regression" and so on. This in the same way as in mathemathics we usually do not define "number", but "natural number", "integers", "real number", "p-adic number" and so on, and if somebody will want to include the quaternions among numbers so be it! it doesn't really matter, what matters is what definitions is used by the book/paper you are reading at the moment.
Definitions are tools, and essentialism, that is discussing what is the essence of ..., what a word really means, are seldom worthwhile.
So, what distinguishes a "regression model" from other kinds of statistical models? Mostly, that there is a response variable, which you want to model as influenced by (or determined by) some set of predictor variables. We are not interested in influence the other direction, and we are not interested in relationships among the predictor variables. Mostly, we take the predictor variables as given, and treat them as constants in the model, not as random variables.
The relationship mentioned above can be linear or nonlinear, specified in a parametric or nonparametric way, and so on.
To delineate from other models we better have a look at some other words often taken to denote something different for "regression models", like "errors in variables", when we accept the possibility of measurement errors in the predictor variables. That could well be included in my description of "regression model" above, but is often taken as an alternative model.
Also, what is meant might vary among fields, see What is the difference between conditioning on regressors vs. treating them as fixed?
To repeat: what matters is the definition used by the authors you are reading now, and not some metaphysics about what it "really is".
Two nice answers were already given, but I'd like to add my two cents.
In regression case we have some random variables $Y$ and $X_1,\dots,X_k$. The variables have some unknown distribution and complicated covariance structure. We simplify this problem to focusing solely on conditional distribution, or more precisely on conditional expectation of $Y$ given the other variables. We simplify it to
$$ \mu = E(y|x_1,\dots,x_k) = f(x_1,\dots,x_k) $$
Where $f$ is a function of predictors that can take different forms (linear, non-linear) depending on particular regression model and $\mu$ is a mean of some distribution when thinking of regression models in terms of generalized linear models. In GLM's $\mu$ can be location of Poisson, Binomial, Gamma etc. distributions. With $L_1$ regularized regression it is a location of Laplace distribution, for robust model minimizing Huber loss so called Huber density is used. In case of quartile regression we focus on other feature of distribution, we estimate $\mu$ that is distribution's quartile rather then expected value.
So instead of looking on full joint distribution, we focus on conditional distribution of $Y$. This simplification is a key feature of regression models.
Some thoughts based on the literature:
F. Hayashi in Chapter 1 of his classic graduate textbook "Econometrics" (2000) states that the following assumptions comprise the classical linear regression model:
Wooldridge in Chapter 2 of his classic introductory econometrics textbook "Introductory Econometrics: A Modern Approach" (2012) states that the following equation defines the simple linear regression model:
$$y=\beta_0+\beta_1 x+u.$$
Greene in Chapter 2 of his popular econometrics textbook "Econometric Analysis" (2011) states
The classical linear regression model consists of a set of assumptions about how a data set will be produced by an underlying “data-generating process.”
and subsequently gives a list of assumptions similar to that of Hayashi's.
Regarding the OP's interest in the GARCH model, Bollerslev "Generalized autoregressive conditional heterosedasticity" (1986) includes a phrase "the GARCH regression model" in the title of section 5 and also in the first sentence of that section. So the father of the GARCH model did not mind calling GARCH a regression model.
Definition and delimitation of regression model
In the past me too shared your perplexity about this point. You refers on econometrics literature and me too refers primarily on that. Unfortunately most econometrics books do not helped much. However I achieved a clearer view that seems me consistent.
What is the definition of a regression model?
It seems me that the correct definition of Regression is as synonym for Conditional Expectation Function (CEF); this definition come from statistical literature. Therefore we can realize that all depends from the joint distribution of the random variables involved $D(y,X)$. The regression is
$E[y|X]=g(X)$
eventually we can speak of regression in error form representation
$y=E[y|X]+\epsilon$
read here for more about that: Regression and the CEF
Frequently people speak about linear regression (model), indeed this is the king object of econometrics. But from previous definition we can realize that linear regression is an explicit specification for $g(X)$. Moreover is possible to show that the true meaning of the famous mean independence assumption $E[\epsilon|X]=0$ is a restriction on $D(y,X)$; it imply linearity for the CEF.
Already here we can realize that assume both linearity for the regression and mean independence for his error is redundant. Worse, most econometrics books speak about Exogeneity assumption in place of mean independence assumption and attribute to it a crucial but strongly different meaning; meaning that cannot be attributed to a regression. Worse again, sometimes this assumption is given in the form $E[\epsilon X]=0$ that is always true by construction in all regressions! This fact reveal undoubtedly the contradiction. Two book the you (Richard Hardy) cited in your reply, Wooldridge 2012 and Greene (2011), are among them.
The core of the problem swing around a conflation among statistical and causal concepts. Indeed exogeneity is, or should be, a causal concept and not an assumption about regression.
What is not a regression model?
A structural equation is not a regression equation (model). The conflation between the two concepts seems me the root of the problems in econometrics literature. Indeed when econometric authors speak about Exogeneity (in any form defined) have in mind something like a structural equation not a regression equation. These my replies go deep about those point:
How would econometricians answer the objections and recommendations raised by Chen and Pearl (2013)?
Under which assumptions a regression can be interpreted causally?
What is the relationship between minimizing prediction error versus parameter estimation error?
moreover other conflation is among the meaning of regression intended as theoretical quantity and properties of his estimator, primarily OLS estimator. For example the “No multicollinearity” assumption is frequently intended as a necessity for uniqueness of OLS estimator, it is an algebraic condition that deal with the data at hand and have few to share with the statistical properties of the random variables involved.
Finally, all above for to said (also) that I do not suggest the use of word "regression" as a meta-concept in the sense suggested by kjetil b halvorsen; Indeed I fear that conflation between regression and structural equation come from that. Indeed concept like conditional quantile function can be interesting but refers on that as quantile regression is (widespread) bad custom. Moreover model like GARCH have much to share with the concept of Skedastic function different from regression one. About model like ARIMA I said that: AR subcase are surely regressions; ARMA is a regression that include unobservable terms; ARIMA looks like a regression but the use of integrated series can bring ad hoc statistical issues.
You can split the question into: (i) "What is a model? and (ii)"What is regression?"
A "model" is any given way of making a prediction, in the sense of "all models are wrong, but some are useful".
"Regression" is the process of adjusting a model with the intention that its predictions become useful.
Typically some class of models is chosen, that is expected (due to some prior knowledge of the problem) to be capable of producing usefully accurate predictions (better than random, and better other models of similar cost and complexity).
Then certain parameters of the model are adjusted to improve its performance, by some measure of performance that is deemed appropriate for the problem the user wishes the model to solve.
See: https://en.wikipedia.org/wiki/Regression_analysis and https://www.investopedia.com/terms/r/regression.asp
Consequently, "What is not a regression model", would be where either or both (i) no prediction is made, (ii) no adjustment is made to improve the prediction. Note that even "descriptive" models still constrain the expected distribution of observations of the thing that they model.
