In the theory of bias-variance decomposition for regression problems (this page is a very nice reference on this theory) noise is defined as $$\mathrm{Noise} = \mathrm{E}_{X,Y}[(Y - \mathrm{E}[Y|X])^2],$$ where $(X,Y)$ is a pair of random variables taken from the given distribution $p$ on $\mathcal{X}\times\mathcal{Y}$ (here $\mathcal{X}$ is a feature space and $\mathcal{Y}$ is a label space). We assume that all observations are generated by $p$ in our regression problem.

How can we reduce noise?
It is easy to show that noise is just a variance of conditional random variable $Y|X$, averaged on $X$: $$\mathrm{Noise} = \mathrm{E}_{X,Y}[(Y - \mathrm{E}[Y|X])^2] = \mathrm{E}_{X}\mathrm{E}_{Y|X}[(Y - \mathrm{E}[Y|X])^2|X] = \mathrm{E}_{X}\mathrm{Var}(Y|X).$$

So, we should decrease this conditional variance (this will lead to noise reduction). Obviously, for a fixed distribution $p(x, y)$, the noise is constant. This means that if someone provides us with the data and prohibits changing it in any way, we simply can't do anything with noise. But if we somehow change our features or labels (for example, generate some new features from the old ones, this is a typical data mining process), we will implicitly move from the original distribution $p(x,y)$ on $\mathcal{X} \times \mathcal{Y}$ to new distribution $\tilde{p}(\tilde{x},\tilde{y})$ on $\tilde{\mathcal{X}} \times \tilde{\mathcal{Y}}$.
My question is – what are the ways to move to this new distribution, so that its noise will be lower that the noise of original distributiion $p(x, y)$.

I see two main ways to do this:

  1. Move to new feature space $\mathcal{X}'$ and new label space $\mathcal{Y}'$, in which the same features and labels are measured more precisely than in original $\mathcal{X}$ and $\mathcal{Y}$ respectively. This is not always possible.
    (If features in $\mathcal{X}$ and/or labels in $\mathcal{Y}$ are measured very inaccurately, then $X$ and $Y$ will be approximately independent, hence $\mathrm{E}[Y|X] \approx \mathrm{E}[Y]$ (where $\mathrm{E}[Y]$ is a population average of $Y$) and noise will be large.)
  2. Move to new, more rich feature space $\tilde{\mathcal{X}}$, which contains more important features than the original feature space $\mathcal{X}$. This is also not always possible.

Are there any other ways to decrease noise in regression problems? Feel free to give any advice.

Note. In many ML texts (for example in the "Elements of Statistical Learning") formula for noise is given for a fixed $X=x_0$, i.e. $\mathrm{Noise} = \mathrm{E}_{Y}[(Y - \mathrm{E}[Y|x_0])^2] = \mathrm{Var}(Y|x_0)$. But in my formulas above I didn't fix $X$ (i.e. I consider more general case).

  • 1
    $\begingroup$ Fixed $x$ has meaning in regression analysis. Quite clear and historical explanations why to do so provide B.Chen and J.Pearl in "Regression and Causation: A Critical Examination of Six Econometrics Textbooks". $\endgroup$ – cure Nov 1 '20 at 23:06
  • $\begingroup$ @Rodvi; You talk about bias-variance tradeoff. It refers on Expected Prediction Error (=EPE). Prediction error can be also translate in “test error”. Your “noise” stand for test error? Are you interested in EPE minimization? $\endgroup$ – markowitz Nov 3 '20 at 12:03
  • $\begingroup$ @markowitz I am interested in improving empirical estimates of the noise term (we can calculate these empirical estimates using finite training sets). I am not interested in general EPE minimization, since it can be done using bias or variance reduction. $\endgroup$ – Rodvi Nov 3 '20 at 12:12
  • $\begingroup$ Well, so written it seem that you are focused on training error. True error is not an “empirical quantity” estimated on a finite sample. So, bring towards zero the training error is an easy task. $\endgroup$ – markowitz Nov 3 '20 at 12:45
  • $\begingroup$ @markowitz sorry, word "training" is unnecessary in my comment, I mean any finite sets from distribution $\tilde p(\tilde x, \tilde y)$ (which are not necessary used for training). Because there is no any learning algorithms at all in the expression for the noise, we can treat it completely independently of them (noise is defined only by distribution). I hope my question post is clear enough to understand that I mean. $\endgroup$ – Rodvi Nov 3 '20 at 13:05

As stated by Dr. Kilian Weinberger, that you mentioned in your question, you can never beat this error.

The optimal classifier gives you the mean of the distribution of all the data P, which you can never get. If you want to detect a car, for example, P has to contain all pictures of cars ever exists.

But if you do find P, there will still some points that vary from its mean. This variation is the noise that cannot be omitted. It defers from one distribution to another.

According to the lecture: error (noise) = E(x,y) = [(y¯(x)−y)^2]). y¯(x) is the label you would expect to obtain from the whole distribution P, given a feature vector x; y is the label that you are testing.

P.S. excuse me for the bad equation, I am new to the platform...

  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Sycorax Nov 3 '20 at 15:37

After some googling I found a great blogpost, whose author (A. Muehlemann, PHD from Oxford) seems to understand noise in the same way as me. I think that his explanation will give your better understanding of my original post.

His conclusion is the following: most of the techniques that reduce bias will also reduce noise (for example, adding some important features will reduce noice, as I said above).

He also said that "In pretty much all practical situations, we find ourselves in the second scenario where we have some (but not all) possible features and thus have some apparent noise. The takeaway message is not that (apparent) “noise” doesn’t exist but rather that “noise” is a misleading term. Perhaps it would be more honest to call it “feature bias”, as it stems from our bias of ignoring parts of reality. In this language, what is classically called “bias” should really be called “modeling bias”, as it stems from our particular choice of one model over another."

In the abovementioned post there is a link to this videolection, where the lecturer made one important assumption about the noise, he modeled $Y$ with additive error model, i.e. $Y=\mathrm{E}[Y|X] + \epsilon(X)$. This is a typical assumption for regression tasks, and using this assumption Noise from my original post can be written as
$\mathrm{Noise} = \mathrm{E}_{X,Y}[(Y - \mathrm{E}[Y|X])^2] = \mathrm{E}_{X, \epsilon}[\epsilon(X)^2]$ and called "stochastic noise".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.