2
$\begingroup$

Been reading some introductory text on statistical modelling and in particular the concepts of reducible and irreducible errors.

As I understand it, reducible error is the bit we can have any hope of minimising by fitting better models or making our models richer by capturing more of the underlying phenomenon.

The thing that confused me was that the author says that the irreducible error could be due to measurement errors or not being able to capture all the features of interest to model the underlying phenomenon in a better way and due to inherent variability in the data. This last phrase has me confused. I thought the underlying variability would arise due to the measurement error. I am not sure I quite understand what is meant by that phrase.

Also, the issue with not being able to capture all the relevant features. Should this not come under reducible errors as if we did measure these features we could produce a better model perhaps?

$\endgroup$
1
  • 8
    $\begingroup$ The heights of men and their weights are associated, but one can be only imperfectly estimated from the other: within any population, the actual weight will vary around from the weight as estimated from the height. Measuring heights to the nearest nanometer and weights to the nearest picogram, even if that were possible, would do nothing to remove this variability. That, I hope, makes it clear that variability and measurement error are not interchangeable concepts. $\endgroup$
    – whuber
    Feb 3, 2017 at 16:44

3 Answers 3

14
$\begingroup$

Assume for the moment that the world is inherently deterministic -- probably a fine approximation if we're operating at levels above the quantum. Any given outcome $y$ is some unknown function of $\mathbf{X}$. What is in $\mathbf{X}$? Everything. We obviously don't observe everything. Hopefully we observe some of the most important things, and the things we miss are not among the most important. But if we don't observe everything that has an effect, then we simply can't find the deterministic relationship between $X$ and $y$. There will always be noise -- which corresponds to the part of $y$ that is determined by things that we don't observe.

So viewed this way, there is no such thing as inherent variability -- inherent variability is just deterministic movement based on factors that you don't observe.

Concretely: you have data on the sales of a hotdog stand, and your $X$'s include features of that hotdog stand. These can explain much of the stand's sales. But features of the stand won't explain everything. Maybe the weather affects hotdogs. Maybe there was a mustard shortage. Maybe Trump tweeted something nasty abut hotdogs. These are deterministic factors that you don't observe, but which could be interpreted as inherent variability.

From this perspective, measurement error is itself a process, which you don't observe. If you knew what caused the error and could predict the error and put it in your model, there would be nothing inherently variable about it.

Finally, if the number of $X$'s is infinite, and your observations of $y$ are finite, you'll never be able to identify the deterministic relationship, even if you know the proper functional form -- which you won't. There will be different combinations of parameters that product identical outcomes with the same structure.

$\endgroup$
3
  • $\begingroup$ Could you elaborate on the last paragraph? Or maybe give a reference if it's something well-known. Thanks. $\endgroup$
    – Leo
    Feb 28, 2017 at 18:24
  • 1
    $\begingroup$ I was referring to the concept of identifiability. I might have played a bit fast and loose however; formalism isn't really my forte. en.wikipedia.org/wiki/Identifiability $\endgroup$ Feb 28, 2017 at 18:34
  • $\begingroup$ @generic_user Why we say that irreducible error sets a lower bound in terms of model's error? I mean the total error can be decomposed into variance + bias + random error. But when variance decrease bias increase and vice versa. So even if one of them is zero the other will not and the total error will be positive (even if random error is zero). $\endgroup$
    – ado sar
    Jul 19, 2022 at 15:21
1
$\begingroup$

whenever we fit the data, we should expect 2 types of errors 1) reducible errors 2) irreducible errors

reducible errors: this is encountered due to either "squared bias" or "squared variance". (refer to bias variance trade-offs for supervised learning for more details)

irreducible errors: it is an error that we cannot reduce by choosing a better model. it is due to randomness or natural variability in a system.

it is denoted as ϵ(Stochastic error term) when fitting a model using equation: Yi = f(Xi) + ϵ

$\endgroup$
1
  • $\begingroup$ This doesn't answer the question. $\endgroup$ Jul 17, 2021 at 8:35
0
$\begingroup$

Irreducible error is not due to noise, but rather due to a randomness or natural variability in systems. It is often also called the aleatoric uncertainty. It may not have a zero mean as such. It cannot be reduced, but can be better characterized. I do not 100% agree with the explanation that the variability arises because of a lack of identifiable feature-set.

Consider you want to make a ML model for predicting density of an Aluminum alloy in a manufacturing plant. Your observations/training sets will look something like this 2700.05, 2701, 2702.995 etc. etc. Nowhere can you attribute this variation to defects in manufacturing processes, but it is a stochasticity inherent to any real systems. You can only characterize this stochasticity better, but you cannot reduce it to zero (or even tending to zero) in limiting cases of perfect models (with all conceivable features) or infinite observations.

$\endgroup$
1
  • $\begingroup$ I think this "inherent stochasticity" you refer to in your example can be seen as a result of several deterministic factors: how the alloy manufacturing equipment works (which can explain the density variance in alloys manufactured by it), time since last maintenance on the equip, years of equip use, variation in building voltage, etc. Therefore, here the variability in the data also arises from the inability to identify all relevant features. As @generic_user said, the world above the quantum level is deterministic, but the number of input features is huge and intractable. $\endgroup$ Jul 17, 2021 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.