Asked a similar question the other day without an answer Link. I think maybe the question there is too big. Here I want to ask a specific one:

  • 2 Class labels (Binary classification labelled with either 1 or 0);
  • 50 Cases (20 labelled as 1 and 30 labelled as 0);
  • 18 Features extracted (some physical quantities);
  • 12 Features remained after feature selection;

How many features can be used for classification to avoid overfitting? Or there is no limit? Some paper mentioned that the number of features should be less than the 1/3 of smallest labelled data. For my example, it should less than (1/3)*20 that is around 7 that means at most 6 or 7 features can be used. Am I right? Thanks a lot.


  • $\begingroup$ I have never seen the comment of the 1/3 of the smallest labeled data. Can you post more info on that. Towards an answer to your question, if the number of dimensions is too large (in relation to N) the data is very likely linearly separable, which makes the problem easy, and there is a higher risk of overfitting. $\endgroup$ – Jacques Wainer Jul 23 '14 at 0:02
  • $\begingroup$ @JacquesWainer Thanks for your reply. There are few publications about this topic. In one paper, the authors proposed an idea of 'sample per feature ratio' link. They mentioned the ratio of 1/3 or even 1/5. However, not sure how general it is. Also, why higher dimension of features tend to be more likely of overfitting? $\endgroup$ – Samo Jerom Jul 23 '14 at 10:27
  • 1
    $\begingroup$ Thanks for the link. If the number of features is too large, the data will be linearly separable and the classification problem is easy - most algorithms will find this separator without "learning anything deep" about the data. But if you dont get a 100% accuracy with new data, then source of the data is not "really" linearly separable - and your algorithms end up using too much information of the training data - overfitting. $\endgroup$ – Jacques Wainer Jul 23 '14 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.