My main question is how could irrelevant features decrease the accuracy?


Lets say we have a feature and the data can be perfectly classified with 100% accuracy using SVM. How can the addition of a new irrelevant feature decrease the accuracy?

Please answer this without example first and then use the example for easy understanding in a 2D plane.

  • 1
    $\begingroup$ Adding an irrelevant feature can only cause overfitting, which will generally lead to decreased accuracy. Which part of this is causing confusion? $\endgroup$ – dsaxton Sep 26 '16 at 16:50
  • 1
    $\begingroup$ This is algorithm-dependent, many will not have decreased accuracy. $\endgroup$ – Firebug Sep 26 '16 at 16:59
  • $\begingroup$ This seems like a bold statement. Which algorithms don't overfit? $\endgroup$ – dsaxton Sep 26 '16 at 17:39
  • $\begingroup$ @dsaxton That isn't the point. Pick 1D linearly separable data, just as OP said. Add a random variable. Train a forest or a maximal margin classifier on both variables. Why do you think always accuracy would change (i.e. be less than 100%)? $\endgroup$ – Firebug Sep 26 '16 at 17:47
  • 2
    $\begingroup$ @Firebug Due to overfitting. It's easy to imagine how this can happen. Say the true boundary is a vertical line through the origin $x_1 = 0$ and now you add another dimension $x_2$. The decision boundary is not going to be a perfect vertical line even though it should be, so for fixed $x_1$ you will make different decisions based on $x_2$ and will be making mistakes when you weren't before. $\endgroup$ – dsaxton Sep 26 '16 at 18:25
  1. With a consistent estimator, your estimate of the irrelevant feature's effect will go to zero as number of observations goes to infinity. In the limit, nothing will happen.
  2. But with finite data, you're giving your algorithm the opportunity to find spurious relationships, to spuriously find that the irrelevant feature does predict your outcome variable.

Including an irrelevant feature gives your estimator new ways to go wrong without any expected benefit. It's like placing a button in a car that should never be pressed; nothing good can be expected to come from it!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.