9
$\begingroup$

Hastie et al. "The Elements of Statistical Learning" (2009) consider a data generating process $$ Y = f(X) + \varepsilon $$ with $\mathbb{E}(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_{\varepsilon}$.

They present the following bias-variance decomposition of the expected squared forecast error at point $x_0$ (p. 223, formula 7.9): \begin{aligned} \text{Err}(x_0) &= \mathbb{E}\left( [ y - \hat f(x_0) ]^2 | X = x_0 \right) \\ &= \dots \\ &= \sigma^2_{\varepsilon} + \text{Bias}^2(\hat f(x_0)) + \text{Var}(\hat f(x_0)) \\ &= \text{Irreducible error} + \text{Bias}^2 + \text{Variance} .\\ \end{aligned} In my own work I do not specify $\hat f(\cdot)$ but take an arbitrary forecast $\hat y$ instead (if this is relevant).
Question: I am looking for a term for $$ \text{Bias}^2 + \text{Variance} $$ or, more precisely, $$ \text{Err}(x_0) - \text{Irreducible error}. $$

$\endgroup$
  • 3
    $\begingroup$ What is the question here? $\endgroup$ – Michael Chernick Apr 12 '17 at 13:19
  • 1
    $\begingroup$ @sntx, thanks for the idea. But it somehow does not sound right. Maybe modelling error (i.e. error due to model misspecification and imprecise estimation of the model), but then it does not make sense if there is no forecast-generating model (e.g. expert forecasts). $\endgroup$ – Richard Hardy Apr 12 '17 at 15:36
  • 1
    $\begingroup$ @DeltaIV, that is rather good. However, I think the term is charged; it seems as if the forecast is poor and we could do better. But suppose we did our best for the given data. So we happen to have chosen the correct model (no "model bias") but the sample is just too small to perfectly estimate the coefficients. The estimation variance ("model variance") is thus really irreducible for the given sample size -- while the term "reducible error" suggests this is not the case. Not that I am sure we can come up with a better term, I still would like to strive for that. $\endgroup$ – Richard Hardy Apr 27 '17 at 11:47
  • 1
    $\begingroup$ @DeltaIV, OK, I now got the intuition in which sense it is reducible. Still the term might be misleading if used without further explanation (just as you had to explain to me). Your latter suggestion is precise, which is really nice, but just as you said, it is quite convoluted. $\endgroup$ – Richard Hardy Apr 27 '17 at 17:45
  • 1
    $\begingroup$ @DeltaIV, I did not intend to sound like that. This is nothing personal; my (hopefully convincing) arguments are above in the comments. But thanks for having the discussion with me, it helps. $\endgroup$ – Richard Hardy Feb 25 '18 at 21:11
4
$\begingroup$

I propose reducible error. This is also the terminology adopted in paragraph 2.1.1 of Gareth, Witten, Hastie & Tibshirani, An Introduction to Statistical Learning, a book which is basically a simplification of ESL + some very cool R code laboratories (except for the fact that they use attach, but, hey, nobody's perfect). I'll list below the reasons the pros and cons of this terminology.


First of all, we must recall that we not only assume $\epsilon$ to have mean 0, but to also be independent of $X$ (see paragraph 2.6.1, formula 2.29 of ESL, 2nd edition, 12th printing). Then of course $\epsilon$ cannot be estimated from $X$, no matter which hypothesis class $\mathcal{H}$ (family of models) we choose, and how large a sample we use to learn our hypothesis (estimate our model). This explains why $\sigma^2_{\epsilon}$ is called irreducible error.

By analogy, it seems natural to define the remaining part of the error, $\text{Err}(x_0)-\sigma^2_{\epsilon}$, the reducible error. Now, this terminology may sound somewhat confusing: as a matter of fact, under the assumption we made for the data generating process, we can prove that

$$ f(x)=\mathbb{E}[Y\vert X=x]$$

Thus, the reducible error can be reduced to zero if and only if $\mathbb{E}[Y\vert X=x]\in \mathcal{H}$ (assuming of course we have a consistent estimator). If $\mathbb{E}[Y\vert X=x]\notin \mathcal{H}$, we cannot drive the reducible error to 0, even in the limit of an infinite sample size. However, it's still the only part of our error which can be reduced, if not eliminated, by changing the sample size, introducing regularization (shrinkage) in our estimator, etc. In other words, by choosing another $\hat{f}(x)$ in our family of models.

Basically, reducible is meant not in the sense of zeroable (yuck!), but in the sense of that part of the error which can be reduced, even if not necessarily made arbitrarily small. Also, note that in principle this error can be reduced to 0 by enlarging $\mathcal{H}$ until it includes $\mathbb{E}[Y\vert X=x]$. In contrast, $\sigma^2_{\epsilon}$ cannot be reduced, no matter how large $\mathcal{H}$ is, because $\epsilon\perp X$.

$\endgroup$
  • $\begingroup$ If noise is the irreducible error, it is not irreducible. You need to motivate this somehow, I cannot do that for myself. $\endgroup$ – Carl Feb 26 '18 at 1:55
  • $\begingroup$ In 2.1.1 the example is "assay of some drug in the blood." The first example I give below is exactly that. In that assay, the so-called irreducible error of measurement is nothing of the kind. It is composed of counting noise, which is usually reduced by counting 10000 or more events, pipetting error, which is almost exponentially distributed, and other technical errors. To further reduce these "irreducible" errors, I recommend using the median of three counting tubes for each time sample. The term irreducible is bad jargon, try again. $\endgroup$ – Carl Feb 26 '18 at 14:32
  • 1
    $\begingroup$ @Delta, thank you for the answer. A one liner "reducible error" might not have been very convincing, but given the context and the discussion it looks pretty good! $\endgroup$ – Richard Hardy Feb 26 '18 at 14:50
  • $\begingroup$ I do not think that the purpose of developing jargon is to confuse people. If you want to say error independent of $n$, versus error that is is function of $n$, say what you mean. $\endgroup$ – Carl Feb 26 '18 at 16:45
  • $\begingroup$ @DeltaV I believe that reducibility is a dubious assumption, see below. $\endgroup$ – Carl Feb 26 '18 at 19:38
0
$\begingroup$

In a system for which all of the physical occurrences have been properly modeled, the left over would be noise. However, there is generally more structure in the error of a model to data than just noise. For example, modelling bias and noise alone do not explain curvilinear residuals, i.e., unmodelled data structure. The totality of unexplained fraction is $1-R^2$, which can consist of misrepresentation of the physics as well as bias and noise of known structure. If by bias we mean only the error in estimating mean $y$, by "irreducible error" we mean noise, and by variance we mean the systemic physical error of the model, then the sum of bias (squared) and systemic physical error is not any special anything, it is merely the error that is not noise. The term (squared) misregistration might be used for this in a specific context, see below. If you want to say error independent of $n$, versus error that is a function of $n$, say that. IMHO, neither error is irreducible, so that the irreducibility property misleads to such an extent that it confuses more than it illuminates.

Why do i not like the term "reducibility"? It smacks of a self-referential tautology as in the Axiom of reducibility. I agree with Russell 1919 that "I do not see any reason to believe that the axiom of reducibility is logically necessary, which is what would be meant by saying that it is true in all possible worlds. The admission of this axiom into a system of logic is therefore a defect ... a dubious assumption."

Below is an example of structured residuals due to incomplete physical modelling. This represents residuals from ordinary least squares fitting of a scaled gamma distribution, i.e., a gamma variate (GV), to blood plasma samples of radioactivity of a renal glomerular filtered radiopharmaceutical [1]. Note that the more data that is discarded ($n=36$ for each time-sample), the better the model becomes so that reducibility deproves with more sample range.

enter image description here

It is notable, that as one drops the first sample at five minutes, the physics improves as it does sequentially as one continues to drop early samples out to 60 min. This shows that although the GV eventually forms a good model for plasma concentration of the drug, something else is going on during early times.

Indeed, if one convolves two gamma distributions, one for early time, circulatory delivery of the drug, and one for organ clearance, this type of error, physical modeling error, can be reduced to less than $1\%$ [2]. Next is an illustration of that convolution.

enter image description here

From that latter example, for a square root of counts versus time graph, the $y$-axis deviations are standardized deviations in sense of Poisson noise error. Such a graph is an image for which errors of fit are image misregistration from distortion or warping. In that context, and only that context, misregistration is bias plus modelling error, and total error is misregistration plus noise error.

$\endgroup$
  • $\begingroup$ Indeed, this is what the above decomposition is about. But your answer would better serve as a comment as it does not address the actual question. Or does it? $\endgroup$ – Richard Hardy Feb 24 '18 at 6:03
  • $\begingroup$ Thanks, but the answer just got further away from the topic. I have a hard time finding any connection between the actual question (how do I call $\text{Bias}^2+\text{Variance}$) and all this... $\endgroup$ – Richard Hardy Feb 24 '18 at 18:56
  • $\begingroup$ Once again, you are answering a different question. A right answer to a wrong question is unfortunately a wrong answer (a note to self: coincidentally, I was explaining this to my undergraduate students yesterday). I am not asking how meaningful the expression is (it is meaningful for someone who has read the ESL textbook and/or worked in applied machine learning), I am asking for a proper term for it. The question is positive, not normative. And it is pretty simple and very concrete. $\endgroup$ – Richard Hardy Feb 24 '18 at 19:58
  • $\begingroup$ @RichardHardy Without the physics, the question was difficult for me to comprehend. Changed my answer, see misregistration above. $\endgroup$ – Carl Feb 24 '18 at 20:11
  • 1
    $\begingroup$ You can do that for estimating the process, yes, and that is the reducible error part. But when you forecast a concrete event that includes the coin flip, there is no way you can reduce the error associated with mispredicting the outcome of the coin flip. This is what the irreducible error is about. Interesting: in a purely deterministic world there would be no irreducible errors by definition, so if your view of the world is completely deterministic, then I might understand what you mean. However, the world is stochastic in "The Elements of Statistical Learning" and in statistics in general. $\endgroup$ – Richard Hardy Feb 26 '18 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.