# In regression analysis what's the difference between data-generation process and model?

In regression analysis what's the difference between 'data-generation process' and 'model'?

• Data-generation process is never known, we choose the model in hope that we approximate the data-generation process sufficiently. That is one of the possible answers, it would help if you provided more context, so it is clearer what kind of answer you are looking for. Check out the chat, currently ongoing journal club discusses article where this issue is raised. Mar 3 '11 at 15:36
• The answers to this question will vary, as they should, because both "data-generation process" and "model" are used in varied ways by various authors. @Weijie, do you have a particular reference in mind?
– whuber
Mar 3 '11 at 15:46

We all have a good sense of what "model" might mean, although its technical definition will vary among disciplines. To compare this to DGP, I began by looking at the top five hits (counting two hits with the same author as one) in Googling "data generation process".

1. A paper on how the US Air Force actually creates data in logistics support.

2. Abstract of a paper published in Environment and Planning A concerning how "synthetic micropopulations" are created via computer "simulation models."

3. A Web page on "synthetic data generation"; that is, simulation "to explore the effects of certain data characteristics on ... models."

4. Abstract of a conference paper in data mining, asserting that "data in databases is the outcome of an underlying data generation process (dgp)."

5. A book chapter that characterizes the data of interest as "arising from some transformation $W_t$ of an underlying [stochastic] process $V_t$ ... some or all [of which] may be unobserved..."

These links exhibit three slightly different but closely related uses of the term "data generation process." The commonest is in a context of statistical simulation. The others refer to the actual means by which data are created in an ongoing situation (logistics) and to a probability model for an ongoing data creation procedure, intended not to be analyzed directly. In the last case the text is differentiating an unobservable stochastic process, which nevertheless is modeled mathematically, from the actual numbers that will be analyzed.

These suggest two slightly different answers are tenable:

1. In the context of simulation or creating "synthetic" data for analysis, the "data generation process" is a way to make data for subsequent study, usually by means of a computer's pseudo random number generator. The analysis will implicitly adopt some model that describes the mathematical properties of this DGP.

2. In the context of statistical analysis, we may want to distinguish a real-world phenomenon (the DGP) from the observations that will be analyzed. We have models for both the phenomenon and the observations as well as a model for how the two are connected.

In regression, then, the DGP would normally describe how a set of data $(\mathbf{X}, Y)_i$ = $(X_{1i}, X_{2i}, \dots, X_{pi}, Y_{i})$, $i=1, 2, \ldots, n$ is assumed to be produced. E.g., the $X_{ji}$ could be set by the experimenter or they could be observed in some way and then be presumed to cause or be related to the values of the $Y_i$. The model would describe the possible ways in which these data could be mathematically related; e.g., we might say that each $Y_{i}$ is a random variable with expectation $\mathbf{X} \mathbf{\beta}$ and variance $\sigma^2$ for unknown parameters $\beta$ and $\sigma$.

• You write the words "cause" or "related". I have a question about this. From your answer seems that DGP concept do not imply causation. However this "relation" is something more than correlation (or any type of association) or not? See also this my related question: stats.stackexchange.com/questions/399671/… Mar 29 '19 at 13:41
• @markowitz "Correlation," strictly speaking, refers to a second moment of the bivariate random variable. I use "related" in the wider sense of "not [statistically] independent."
– whuber
Mar 29 '19 at 13:49
• I know, and exactly for this reason I stated "or any type of [only statistical] association". Can I repeat my question as: However is this "relationship" something more than the association or not? Starting from the concept of "true model", sometimes used as a synonym of DGP, it seems something more. If so, I don't understand exactly what it is. My previous link give an example. Mar 29 '19 at 14:15
• @markowitz I'm afraid I don't understand what you're trying to ask. That may be because I'm not sure what you mean precisely by "relationship" or "association." I did look at your link, but the unusual English usage doesn't convey anything meaningful to me.
– whuber
Mar 29 '19 at 14:40
• I'm sorry for my English. I tried to modify the linked question in clearer sense. I hope that it is understandable. Mar 29 '19 at 15:33

Whuber's answer is excellent, but it is worth adding emphasis to the fact that a statistical model need not resemble the data generating model in every respect to be an appropriate model for inferential exploration of data. Liu and Meng explain that point with great clarity in their recent arXived paper (http://arxiv.org/abs/1510.08539):

Misconception 1. A probability model must describe the generation of the data.

A more apt description of the model’s job (in inference) is “Such and such probabilistic pattern produces data which resemble ours in important ways.” To create replicas (i.e., controls) of the Mona Lisa, one does not need to bring da Vinci back to life — a camera and printer will suffice for most purposes. Of course, knowledge of da Vinci’s painting style will improve the quality of our replicas, just as scientific knowledge of the true data generating process helps us design more meaningful controls. But for purposes of uncertainty quantification, our model’s job is to specify a set of controls that resemble (D,$\theta$). Nowhere is this point clearer than in applications involving computer experiments where a probabilistic pattern is used to describe data following a known (but highly complicated) deterministic pattern (Kennedy and O’Hagan, 2001; Conti et al., 2009). We need a descriptive model, not necessarily a generative model. See Lehmann (1990), Breiman (2001) and Hansen and Yu (2001) for more on this point.

• +1. I especially like the distinction between descriptive and generative models of data.
– whuber
Sep 18 '19 at 13:49

The DGP is the true model. The model is what we have tried to, using our best skills, to represent the true state of nature. The DGP is influenced by "noise". Noise can be of many kinds:

1. One time interventions
2. Level shifts
3. Trends
4. Changes in Seasonality
5. Changes in Model Parameters
6. Changes in Variance

If you don't control for these 6 items than your ability to identify the true DGP is reduced.

DGP is the virtual reality and a unique recipe for simulation. A model is a collection of DGP or possible ways that the data could have been generated.

Read the first page of this mini course by Russell Davidson:

http://russell-davidson.arts.mcgill.ca/Aarhus/bootstrap_course.pdf