Does the irreducible error in a model include "unknown" variables?

Question

My informal understanding of the irreducible error for a set of data that no algorithm can avoid is essentially the "noise" i.e the purely random effects.

As part of refining/honing that understanding I am reading A Gentle Introduction to Bias Variance Tradeoff https://machinelearningmastery.com/gentle-introduction-to-the-bias-variance-trade-off-in-machine-learning/ that makes the following statement:

The irreducible error cannot be reduced regardless of what algorithm is used. It is the error introduced from the chosen framing of the problem and may be caused by factors like unknown variables that influence the mapping of the input variables to the output variable.

This statement seems to diverge from my prior understanding - in effect saying that systemic disparities between the observed and predicted results may be considered irreducible. I would contend that any such non-random differences constitute bias.

Would a seasoned statistician be able to comment on either

1. the wording of that paragraph above were either misleading or flat-out incorrect or
2. the paragraph is phrased accurately .

If (2) then a further clarification of that statement in the context of random vs systemic errors would be appreciated.

Answer 1

You are right that the wording seems strange. Say the true model is $$ Y=\alpha+\beta x + \gamma z+\epsilon^* $$ but you did not measure $z$, so the effective error term is $\epsilon=\epsilon^*+\gamma z$. Then if $z$ is correlated with $x$, calculate $$ \DeclareMathOperator{\C}{\mathbb{C}} \C(\epsilon,x)=\gamma\C(z,x) $$ so nonzero. So in this case, the error term is not independent of the variables in the model. This could indeed show itself as bias.

The question is related to what is called omitted variable bias. this blog link is interesting.

Answer 2

I do not think there is a clear cut answer to your title question because of some difficulties with defining the information set with regards to which the decomposition of the mean squared prediction error is done. In a sense, it depends on where we draw the line. Take a trivial data generating process $$ y=\beta_0+\beta_1 x+\varepsilon $$ as an example. On the one hand, an all-seeing power who can observe $y$, $x$ and $\varepsilon$ would consider this a deterministic process. It would draw a line between $(y,x,\varepsilon)$ on the one side and nothing on the other side. Relative to this line, the irreducible error is zero. On the other hand, an ordinary person who can only observe $y$ and $x$ but not $\varepsilon$ would consider this a stochastic process. It would draw a line between $(y,x)$ on th one side and $\varepsilon$ on the other. Relative to this line, the irreducible error is $\text{Var}(\varepsilon)=\sigma^2_{\varepsilon}$. So it depends on the information set, on the universe of variables that are considered observable.

When the all-seeing power looks at the perspective of the ordinary person, it sees a concrete variable ($\varepsilon$) lurking on the "wrong" side of the line, and the variable appears "systemic", so the power would like to bring it to the "correct" side and drive the irreducible error to zero. Meanwhile, the ordinary person can also imagine having the perspective of the all-seeing power, but it remains a dream as there is no way to observe $\varepsilon$ and bring it to the other side of the line, hence $\varepsilon$ can be called "irreducible".

Here is the clash; on the one hand, there are "systemic" variables within $\varepsilon$, but on the other hand, the irreducible error remains, well, irreducible w.r.t. the information set of the ordinary person.

So I would disagree with

the wording of that paragraph above were either misleading or flat-out incorrect or

(at least the flat-out wrong part) though I hesitate to endorse the claim

the paragraph is phrased accurately

without some qualifications such as the above (and below).

When modelling real world processes, you can rarely be sure you have extracted everything from the $\varepsilon$ side of the line so that you can claim $y=f(X)+\varepsilon$ where $\varepsilon$ is entirely irreducible. At the same time, for many processes you know you will never be able to reduce $\varepsilon$ all the way to zero. So you draw the line some place, and then the decomposition of the mean squared prediction error is done with respect to that line.

---

For the part unknown variables that influence the mapping of the input variables to the output variable, you could have $$ y=\delta_0+\delta_1 x+\delta_2 xz+\varepsilon $$ with $z$ unobservable to the regular person and independent of $x$ and $\varepsilon$ with possible values $\{0,1\}$ with probability $0.5$ each. The discussion above should apply here about as well as for the initial model.

---

_{Another thought: If you view the world as completely deterministic (even though many of the phenomena are too complex to comprehend), then there can be no irreducible errors. If you think there are some genuinely stochastic elements in the world, then they are the causes of the irreducible errors. In any case, there likely are many examples where the irreducible error is very close to zero. However, we cannot get close to it in practice because of the limited number of variables that we choose to observe, limited sample sizes and limited computational power.}