Noise in regression problems and ways to reduce it

Question

In the theory of bias-variance decomposition for regression problems (this page is a very nice reference on this theory) the noise is defined as $$\mathrm{Noise} = \mathrm{E}_{X,Y}[(Y - \mathrm{E}[Y|X])^2],$$ where $(X,Y)$ is a pair of random variables taken from the given distribution $p$ on $\mathcal{X}\times\mathcal{Y}$ (here $\mathcal{X}$ is a feature space and $\mathcal{Y}$ is a label space). We assume that all observations are generated by $p$ in our regression problem.

How can we reduce the noise?
It is easy to show that the noise is just a conditional variance $\mathrm{Var}(Y|X)$, averaged on $X$: $$\mathrm{Noise} = \mathrm{E}_{X,Y}[(Y - \mathrm{E}[Y|X])^2] = \mathrm{E}_{X}\mathrm{E}_{Y|X}[(Y - \mathrm{E}[Y|X])^2|X] = \mathrm{E}_{X}\mathrm{Var}(Y|X).$$

So, we should decrease this conditional variance (this will lead to noise reduction). Obviously, for a fixed distribution $p(x, y)$, the noise is a constant. This means that if someone provides us with the data and prohibits changing it in any way, we simply can't do anything with the noise. But if we somehow change our features or labels (for example, generate some new features from the old ones, this is a typical data mining process), we will implicitly move from the original distribution $p(x,y)$ on $\mathcal{X} \times \mathcal{Y}$ to new distribution $\tilde{p}(\tilde{x},\tilde{y})$ on $\tilde{\mathcal{X}} \times \tilde{\mathcal{Y}}$.
My question is – what are the ways to move to this new distribution, so that its noise will be lower that the noise of original distributiion $p(x, y)$.

I see two main ways to do this:

1. Move to new feature space $\mathcal{X}'$ and new label space $\mathcal{Y}'$, in which the same features and labels are measured more precisely than in original $\mathcal{X}$ and $\mathcal{Y}$ respectively. This is not always possible.
   (If features in $\mathcal{X}$ and/or labels in $\mathcal{Y}$ are measured very inaccurately, then $X$ and $Y$ will be approximately independent, hence $\mathrm{E}[Y|X] \approx \mathrm{E}[Y]$ (where $\mathrm{E}[Y]$ is a population average of $Y$) and noise will be large.)
2. Move to new, more rich feature space $\tilde{\mathcal{X}}$, which contains more important features than the original feature space $\mathcal{X}$. This is also not always possible.

Are there any other ways to decrease noise in regression problems?

---

Note. In many ML texts (for example in the "Elements of Statistical Learning") formula for the noise is given for a fixed $X=x_0$, i.e. $\mathrm{Noise} = \mathrm{E}_{Y}[(Y - \mathrm{E}[Y|x_0])^2|x_0] = \mathrm{Var}(Y|x_0)$. But in my formulas above I didn't fix $X$ (i.e. I considered more general case).

Answer 1

As stated by Dr. Kilian Weinberger, that you mentioned in your question, you can never beat this error.

The optimal classifier gives you the mean of the distribution of all the data P, which you can never get. If you want to detect a car, for example, P has to contain all pictures of cars ever exists.

But if you do find P, there will still some points that vary from its mean. This variation is the noise that cannot be omitted. It defers from one distribution to another.

According to the lecture: error (noise) = E(x,y) = [(y¯(x)−y)^2]). y¯(x) is the label you would expect to obtain from the whole distribution P, given a feature vector x; y is the label that you are testing.

P.S. excuse me for the bad equation, I am new to the platform...

Answer 2

After some googling I found a great blogpost, whose author (A. Muehlemann, PHD from Oxford) seems to understand noise in the same way as me. I think that his explanation will give your better understanding of my original post.

His conclusion is the following: most of the techniques that reduce bias will also reduce noise (for example, adding some important features will reduce noice, as I said above).

He also said that "In pretty much all practical situations, we find ourselves in the second scenario where we have some (but not all) possible features and thus have some apparent noise. The takeaway message is not that (apparent) “noise” doesn’t exist but rather that “noise” is a misleading term. Perhaps it would be more honest to call it “feature bias”, as it stems from our bias of ignoring parts of reality. In this language, what is classically called “bias” should really be called “modeling bias”, as it stems from our particular choice of one model over another."

---

In the abovementioned post there is a link to this videolection, where the lecturer made one important assumption about the noise, he modeled $Y$ with additive error model, i.e. $Y=\mathrm{E}[Y|X] + \epsilon(X)$. This is a typical assumption for regression tasks, and using this assumption Noise from my original post can be written as
$\mathrm{Noise} = \mathrm{E}_{X,Y}[(Y - \mathrm{E}[Y|X])^2] = \mathrm{E}_{X, \epsilon}[\epsilon(X)^2]$ and called "stochastic noise".