2
$\begingroup$

My informal understanding of the irreducible error for a set of data that no algorithm can avoid is essentially the "noise" i.e the purely random effects.

As part of refining/honing that understanding I am reading A Gentle Introduction to Bias Variance Tradeoff https://machinelearningmastery.com/gentle-introduction-to-the-bias-variance-trade-off-in-machine-learning/ that makes the following statement:

The irreducible error cannot be reduced regardless of what algorithm is used. It is the error introduced from the chosen framing of the problem and may be caused by factors like unknown variables that influence the mapping of the input variables to the output variable.

This statement seems to diverge from my prior understanding - in effect saying that systemic disparities between the observed and predicted results may be considered irreducible. I would contend that any such non-random differences constitute bias.

Would a seasoned statistician be able to comment on either

  1. the wording of that paragraph above were either misleading or flat-out incorrect or
  2. the paragraph is phrased accurately .

If (2) then a further clarification of that statement in the context of random vs systemic errors would be appreciated.

$\endgroup$

2 Answers 2

1
$\begingroup$

You are right that the wording seems strange. Say the true model is $$ Y=\alpha+\beta x + \gamma z+\epsilon^* $$ but you did not measure $z$, so the effective error term is $\epsilon=\epsilon^*+\gamma z$. Then if $z$ is correlated with $x$, calculate $$ \DeclareMathOperator{\C}{\mathbb{C}} \C(\epsilon,x)=\gamma\C(z,x) $$ so nonzero. So in this case, the error term is not independent of the variables in the model. This could indeed show itself as bias.

The question is related to what is called omitted variable bias. this blog link is interesting.

$\endgroup$
7
  • $\begingroup$ You are comparing models $$y=\alpha+\beta x+\gamma z+\epsilon^*$$ and $$y=\alpha+\beta x+\epsilon,$$ but who would use the second model (one with $\epsilon$ defined as $\epsilon=\gamma z+\epsilon^*)$? Does there exist a consistent estimator of $\beta$ based on $y$ and $x$ alone? I do not think so. The model that would actually be used when $z$ is unobserved is $$y=\alpha+\delta x+u$$ with $u$ independent of $x$. $\endgroup$ Apr 19, 2020 at 11:44
  • $\begingroup$ Yes, that is the modell that would be used, bit IT would be wrong, because u Is not independent of x $\endgroup$ Apr 19, 2020 at 12:11
  • $\begingroup$ Cannot we choose $\delta$ to yield $x$ independent of $u$? I think OLS would be consistent estimator of such a $\delta$. And now I noticed that $\alpha$ in my last model should be something else, say, $\delta_0$. Unrelated note: I do not think $z$ in your model influences the mapping of $x$ to $y$. If so, it is not an ideal example for the quote. $\endgroup$ Apr 19, 2020 at 12:58
  • 1
    $\begingroup$ Difficult to think about indeed! You are right, $u$ is not independent of $x$, just uncorrelated with it (and $x$ is "presistent" / predictively consistent). Also note that bias of an estimator and bias of a model (as in this question) are in a sense separate things (though not unrelated). You demonstrate bias of an estimator, but the relevant bias here is that of the model. And depending on whether you include $z$ in the information set, the model is either biased or not (as I tried to articulate in my answer). $\endgroup$ Apr 20, 2020 at 5:50
  • 1
    $\begingroup$ Would the definition given here (citing Hastie et al. "Elements of Statistical Learning") suffice? I might have formulated myself poorly, but this is what I most likely have meant (this was months ago). $\endgroup$ Sep 9, 2020 at 17:28
2
$\begingroup$

I do not think there is a clear cut answer to your title question because of some difficulties with defining the information set with regards to which the decomposition of the mean squared prediction error is done. In a sense, it depends on where we draw the line. Take a trivial data generating process $$ y=\beta_0+\beta_1 x+\varepsilon $$ as an example. On the one hand, an all-seeing power who can observe $y$, $x$ and $\varepsilon$ would consider this a deterministic process. It would draw a line between $(y,x,\varepsilon)$ on the one side and nothing on the other side. Relative to this line, the irreducible error is zero. On the other hand, an ordinary person who can only observe $y$ and $x$ but not $\varepsilon$ would consider this a stochastic process. It would draw a line between $(y,x)$ on th one side and $\varepsilon$ on the other. Relative to this line, the irreducible error is $\text{Var}(\varepsilon)=\sigma^2_{\varepsilon}$. So it depends on the information set, on the universe of variables that are considered observable.

When the all-seeing power looks at the perspective of the ordinary person, it sees a concrete variable ($\varepsilon$) lurking on the "wrong" side of the line, and the variable appears "systemic", so the power would like to bring it to the "correct" side and drive the irreducible error to zero. Meanwhile, the ordinary person can also imagine having the perspective of the all-seeing power, but it remains a dream as there is no way to observe $\varepsilon$ and bring it to the other side of the line, hence $\varepsilon$ can be called "irreducible".

Here is the clash; on the one hand, there are "systemic" variables within $\varepsilon$, but on the other hand, the irreducible error remains, well, irreducible w.r.t. the information set of the ordinary person.

So I would disagree with

the wording of that paragraph above were either misleading or flat-out incorrect or

(at least the flat-out wrong part) though I hesitate to endorse the claim

the paragraph is phrased accurately

without some qualifications such as the above (and below).

When modelling real world processes, you can rarely be sure you have extracted everything from the $\varepsilon$ side of the line so that you can claim $y=f(X)+\varepsilon$ where $\varepsilon$ is entirely irreducible. At the same time, for many processes you know you will never be able to reduce $\varepsilon$ all the way to zero. So you draw the line some place, and then the decomposition of the mean squared prediction error is done with respect to that line.


For the part unknown variables that influence the mapping of the input variables to the output variable, you could have $$ y=\delta_0+\delta_1 x+\delta_2 xz+\varepsilon $$ with $z$ unobservable to the regular person and independent of $x$ and $\varepsilon$ with possible values $\{0,1\}$ with probability $0.5$ each. The discussion above should apply here about as well as for the initial model.


Another thought: If you view the world as completely deterministic (even though many of the phenomena are too complex to comprehend), then there can be no irreducible errors. If you think there are some genuinely stochastic elements in the world, then they are the causes of the irreducible errors. In any case, there likely are many examples where the irreducible error is very close to zero. However, we cannot get close to it in practice because of the limited number of variables that we choose to observe, limited sample sizes and limited computational power.

$\endgroup$
2
  • 2
    $\begingroup$ I find this question very interesting and have spent some time thinking about it. Obviously my thoughts have not matured into a short, elegant answer. I nevertheless hope my consideration can be helpful, and I hope to be able to improve them over time. $\endgroup$ Apr 19, 2020 at 20:23
  • $\begingroup$ I read (and upvoted) when you posted earlier in the day - but have not completely absorbed it yet. $\endgroup$ Apr 20, 2020 at 4:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.