reducible vs irreducible error

Question

Been reading some introductory text on statistical modelling and in particular the concepts of reducible and irreducible errors.

As I understand it, reducible error is the bit we can have any hope of minimising by fitting better models or making our models richer by capturing more of the underlying phenomenon.

The thing that confused me was that the author says that the irreducible error could be due to measurement errors or not being able to capture all the features of interest to model the underlying phenomenon in a better way and due to inherent variability in the data. This last phrase has me confused. I thought the underlying variability would arise due to the measurement error. I am not sure I quite understand what is meant by that phrase.

Also, the issue with not being able to capture all the relevant features. Should this not come under reducible errors as if we did measure these features we could produce a better model perhaps?

Answer 1

Assume for the moment that the world is inherently deterministic -- probably a fine approximation if we're operating at levels above the quantum. Any given outcome $y$ is some unknown function of $\mathbf{X}$. What is in $\mathbf{X}$? Everything. We obviously don't observe everything. Hopefully we observe some of the most important things, and the things we miss are not among the most important. But if we don't observe everything that has an effect, then we simply can't find the deterministic relationship between $X$ and $y$. There will always be noise -- which corresponds to the part of $y$ that is determined by things that we don't observe.

So viewed this way, there is no such thing as inherent variability -- inherent variability is just deterministic movement based on factors that you don't observe.

Concretely: you have data on the sales of a hotdog stand, and your $X$'s include features of that hotdog stand. These can explain much of the stand's sales. But features of the stand won't explain everything. Maybe the weather affects hotdogs. Maybe there was a mustard shortage. Maybe Trump tweeted something nasty abut hotdogs. These are deterministic factors that you don't observe, but which could be interpreted as inherent variability.

From this perspective, measurement error is itself a process, which you don't observe. If you knew what caused the error and could predict the error and put it in your model, there would be nothing inherently variable about it.

Finally, if the number of $X$'s is infinite, and your observations of $y$ are finite, you'll never be able to identify the deterministic relationship, even if you know the proper functional form -- which you won't. There will be different combinations of parameters that product identical outcomes with the same structure.

Answer 2

whenever we fit the data, we should expect 2 types of errors 1) reducible errors 2) irreducible errors

reducible errors: this is encountered due to either "squared bias" or "squared variance". (refer to bias variance trade-offs for supervised learning for more details)

irreducible errors: it is an error that we cannot reduce by choosing a better model. it is due to randomness or natural variability in a system.

it is denoted as ϵ(Stochastic error term) when fitting a model using equation: Yi = f(Xi) + ϵ

Answer 3

Irreducible error is not due to noise, but rather due to a randomness or natural variability in systems. It is often also called the aleatoric uncertainty. It may not have a zero mean as such. It cannot be reduced, but can be better characterized. I do not 100% agree with the explanation that the variability arises because of a lack of identifiable feature-set.

Consider you want to make a ML model for predicting density of an Aluminum alloy in a manufacturing plant. Your observations/training sets will look something like this 2700.05, 2701, 2702.995 etc. etc. Nowhere can you attribute this variation to defects in manufacturing processes, but it is a stochasticity inherent to any real systems. You can only characterize this stochasticity better, but you cannot reduce it to zero (or even tending to zero) in limiting cases of perfect models (with all conceivable features) or infinite observations.