2
$\begingroup$

Often, we try to model data $ \{(x_i, y_i): \ i=1,2,...,n \}$ by assuming a "data generating process" for the data: $$Y = f(x)+\epsilon$$

where we call $f(x)$ the (deterministic) "systematic component" and $\epsilon$ the (random) "non-systematic component."

I don't quite understand what these terms mean. $Y$ is supposed to be random and $x\in \mathbb R^p$ is supposed to be fixed and set by the experimenter.

Could someone please explain this equation and define "data generating process," "systematic component" and "non-systematic component"? Can we always decompose a random variable into something "systematic" and "random"? Is a "data generating process" the same thing as a "statistical model," and is that the same as a "structural equation" or a "theory equation"?

What does systematic mean in this context? What can be systematic/deterministic about a random variable $Y$?

Sometimes, $f(x)$ is called the regression function $E(Y|x)$ when we assume the random errors have mean zero $E(\epsilon|x)=0$.

I don't quite understand what this notation $E(Y|x)$ means. If we define the predictors $x$ to be deterministic and set by the experimenter, how can one condition on a deterministic scalar? What is the definition of this?

This is partially with reference to the Elements of Statistical Learning: https://web.stanford.edu/~hastie/ElemStatLearn//printings/ESLII_print12.pdf

(Sorry for so many questions, but they are all related to the given equation and I think it would be amazing if someone can provide an answer which incorporates all of these requested clarifications)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Let me rephrase your questions and (try) to answer :

As $\mathbb{E}[Y\mid x]$ is short notation for $\mathbb{E}[Y\mid X = x]$ it assumes existence of a random variable $X$. How to make sense of this when $x$ is a fixed variable (by experiment designer for instance) ?

  • This expression is only useful when there are several possible values for $x$ and you could imagine a random variable $X$ which can take these different values at random with uniform probability. Then the expression makes sense. The fact is that $\mathbb{E}[Y\mid X = x]$ does not depend on the distribution of $X$, and you could imagine infinitely many different random variables whose possible values are the $x$ values, so we don't really care what the random variable is (or even if it is really random), since the only important feature of $X$ is its support. Maybe it is more convenient to see this as an abuse of notation for $\mathbb{E}[Y(x)]$ in a case where random variable $Y$ depends on a deterministic parameter $x$ and $Y = Y(x)$.

What is a "data generating process"?

  • I see it as a synonym of model (maybe someone would correct me...). So this is assumed by the data analyst. It would be the way one would take to mimic nature in the procuction of data from systematic variables. More formally and using the same notation as you, this would be a function that takes as input systematic variables $x$ and output (eventually) random variables $Y$.

What are "systematic component" and "random component"?

  • It must be clear that one can only talk about systematic and random component of a data generating process (or model), not of a random variable. This is a decomposition of a model into a random part and a deterministic part. Such decomposition of models may have no deterministic component, for example: $Y = \varepsilon $ where $\varepsilon \sim \mathcal{N}(0, 1)$, and some may have no random component, for example : $Y = 2 \times x$. Actually this decomposition is not unique. The first model $Y = \varepsilon$ can also be written $Y = x + \eta$ where $\eta \sim \mathcal{N}(-x, 1)$. So "deterministic component" and "random component" refers to components of a decomposition of a model, usually the simplest one, or one that assumes some condition on one of the components (e.g. linear systematic component part or centred i.i.d. random component). There is no standard way of defining such a decomposition. In the example you show, it is in the sense of summation, but if could be multiplicative or even more complex.

Can we always decompose a random variable into something "systematic" and "random"?

  • Well, systematic and random components are a decomposition of a data generating process not of a random variable. So the right question would be "can we always find a data generating process that generates $Y$"? And the answer is yes, but a simple and usefull one.. not sure.

I hope it does answer some of your interrogations..

Improve this answer
$\endgroup$
2
  • $\begingroup$ 'I see it as a synonym of model (maybe someone would correct me...)' - I'm currently looking for systematic references to DGP, but am finding it hard to get a authoritative one, just scraps here and there as popped into my answer. My understanding is that the model is a hypothesis of what the underlying DGP is. The DGP is what caused our observations to happen as they did, but we can only hypothesis then test what that might have been. $\endgroup$
    – ReneBt
    Sep 21, 2020 at 14:05
  • $\begingroup$ Hey @Pohoua. Thank you very much for this reply. This clarified a lot for me. Thank you for your time! (Specifically, the difference between a model for the data-generating process vs. actually decomposing a random variable! And also how we can think of E(Y|x) as E(Y(x))) $\endgroup$
    – user523384
    Oct 16, 2020 at 8:02
1
$\begingroup$

Data Generating Process

A data generating process is the process that generates the observed data. We do not KNOW it, but we can make hypotheses about it.

Statistical Model

A statistical model is a hypothesis about that process and we test the model against the observed data to determine how well it fits. If we are comfortable with the degree of fit we consider our hypothesis to have become a theory.

here describes the order of observation and DGP

The types of the data sets are not determined by the visualization process, but by the data generation process

In this case ϵ becomes the residual unaccounted for by the model. It is misleading to label it unsystematic or noise because in most cases what it contains is lower order processes contributing to your desired variation (signal) and contributing to undesired variation (noise). If you dismiss it as noise you will never refine your hypothesis and update your model. If you recognise that it may contain some real processes you hadn't anticipated you can explore it and deepen your understanding.

If the universe is truly deterministic then there is no such thing as random and even the tiniest blip in the DGP is non-random. Rather, it may be caused by the faint ghosts of quantum entanglement just after a particle condensed from the big bang, propagated over billions of years and diluted by interactions with other particles and fields. If quantum mechanics has truly random elements then you can push the DGP back to stochastic processes on a quantum level.

Here the authors discuss randomness in exactly this kind of ambiguous way, as a result of deterministic processes

Randomness and data imperfection are two direct consequences of the dynamic nature of stream data. There could be several unforeseeable factors that affect the processing chain. For example, the data generation process may induce randomness because the data sources are normally independently installed in different environments, which makes it nearly impossible to guarantee the sequence of data arrival across different streams

So then for the specific items requested:

"systematic component"

this is the desired variation, often referred to as signal. It is something that can be described succinctly and systematically.

"non-systematic component"?

this is any variation that lies outside the specified hypothesis. In a deterministic world true randomness is impossible, but is used as shorthand for stuff that is too complicated to untangle. In a world containing randomness the component will not just be the original noise, but all the events that it has propagated into.

Can we always decompose a random variable into something "systematic" and "random"?

As Pohoua says, this is confusing the terminology -a random variable can be combined with a systematic process in a stochastic data generating process. A truly random variable would have zero systematic contributions, something we can't generate.

Is a "data generating process" the same thing as a "statistical model," and is that the same as a "structural equation" or a "theory equation"?

See above for first part (No). A structural equation (or theory equation) is usually the terminology used when a mathematical model is generated based on theory rather than data and is then fitted to the observed data to test. Here comparing physical models to the DGP is mentioned.

If we know something about the physics of the data generation process, we can use that information to construct a model

Sometimes structured equation modelling is used in the context of regression as it creates a structured equation through statistical modelling, but many don't like this usage.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Hey @ReneBt. Thank you very much for your reply. This was very helpful, and really helped clarify things for me! $\endgroup$
    – user523384
    Oct 16, 2020 at 8:00
  • $\begingroup$ I had one question, if that's okay. Even though we say a "truly" random variable cannot have a systematic component, why is it that we model it as so? What would you call something like $Y(x) = f(x) + \epsilon$, being a "mixture" of a deterministic function and a random variable $\endgroup$
    – user523384
    Oct 16, 2020 at 9:03
  • 1
    $\begingroup$ It is a pragmatic approach. What is 'truly' random is a deep philosophical question and if we waited till we were satisfied what it meant then we would never develop useful models of the world. Whether anything is ever 'truly' random depends on whether the 'randomness' we observe is the result of deterministic events beyond our measure, or whether the universe does indeed experience non-deterministic events. If you go down that rabbit hole you make interesting philosophical progress but it could be a long time before it makes a practical impact on a problem we are being asked to solve ASAP. $\endgroup$
    – ReneBt
    Oct 16, 2020 at 9:54
  • $\begingroup$ I understand, so in a way, we use the notation $Y|x$ to mean "after we use the information that $x$ is the predictor, $Y$ is a random variable distributed around it?" Mathematically it might be nice to define what this means. I've heard that $x$ is called a "hyperparameter" $\endgroup$
    – user523384
    Oct 16, 2020 at 10:03
  • 1
    $\begingroup$ The joy of bumping word definitions into Math. On one hand we have words explaining math results that mean something different to everyone. On the other we have abstract notation that is much more rigorously defined, but ultimately still described with words. Currently I am on a quest to pin down what statisticians mean by 'data generating process' and seem to see subtly different implications in every source I read, including my own take. I digress. $Y(x)-f(x) = \epsilon$ the $\epsilon$ residual after accounting for $x$ is the 'random' part, $Y$ is a linear combination of $x$ and $\epsilon$ $\endgroup$
    – ReneBt
    Oct 16, 2020 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.