I have a small data set (under 100 samples) and 5 input features. I often hear about how neural networks are prone to overfitting under such conditions and that naive Bayes is likely to underfit.

What approach should I try then ? I should mention that the chosen classifier must return the probability/likelihood of belonging to a certain class and not just return the class without any provided uncertainty.

  • 1
    $\begingroup$ Logistic regression is my go to first choice. $\endgroup$ – Matthew Drury Apr 3 '17 at 0:00
  • $\begingroup$ With such a small dataset, any difference in the performance of various approaches will be essentially within the noise. So there is little use in trying a bunch of different approaches and taking the best. As @Matthew noted, logistic regression is a good default approach to try. Neural networks need a lot of data to work well, and naive Bayes is not used in modern ML any more (it's more of a teaching example these days). $\endgroup$ – AaronDefazio Apr 3 '17 at 1:51
  • $\begingroup$ Agree with @AaronDefazio, you absolutely must rigorously bootstrap all of your performance measures when tuning your learning algorithms. It is very likely that differences in performance point estimates will be due to chance. $\endgroup$ – Matthew Drury Apr 3 '17 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.