Studying the bias-variance trade-off:

expected loss = bias + variance + noise

I understand that we minimize this quantity by finding the "best" balance between low bias/high variance and high bias/low variance. However, the noise term is beyond our control. So in a sense, if noise is large, then learning is pointless, right? Are there techniques for detecting when this might be the case?


When noise is "large" then learning is not pointless, but it's "expensive" in some sense. For instance, you know the expression "house always wins". It means that the odds favor the casino against the gambler. However, the odds can be very close to 1:1, they may only so slightly be tilted towards the "house", e.g. 0.5%. Hence, you may call the outcomes series very noisy in some cases, yet the casinos make a ton of money in a long run. So, the fact that the data is "noisy" doesn't mean in isolation that the learning will be pointless or useless or unprofitable.

  • $\begingroup$ Consider also that a lot of "machine learning algorithms" are statistical models or equivalent to statistical models that have uses beyond making predictions with minimal error $\endgroup$ – shadowtalker May 27 '15 at 15:39
  • $\begingroup$ I see. So you are saying that although it may be "pointless" to try to predict based on data coming from fair coin flips, it may still be very worth it if the coin is even only slightly biased? $\endgroup$ – Fequish May 28 '15 at 12:27
  • $\begingroup$ It may or may not be pointless depending on the purpose of modeling. You want low noise, but sometimes even with large noise there's something to gain from the model. Consider the weather forecast. It's very noisy, so I wouldn't subscribe to any paid weather service. However I look at it every morning because it's free, costs me nothing and in a long run I benefit from it. $\endgroup$ – Aksakal May 28 '15 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.