1
$\begingroup$

I am using perceptron to solve a classification problem. I have a limited amount of features (26) and iterate through all possible combinations of them. A combination of two features [feature_a, feature_b] results in better accuracy than the same combination with additional third feature [feature_a, feature_b, feature_c]. Is there any reasonable explanation for that apart from a bug in my code?

I tried replacing perceptron by naive bayes classifiers, the pattern persists.

I am using scikit-learn Python library for perceptron and NB implementations, if that's important.

$\endgroup$
1
$\begingroup$

I think that is the problem of overfitting. Adding features can always improve training acc. in such a shallow model, but may hurt generalization.

p.s., I think you may try logsitic regression or linear SVM rather than perception.

$\endgroup$
  • $\begingroup$ I thought overfitting would occur if I had much more features than 3, is it possible with such a simple model? $\endgroup$ – user3399516 Apr 22 '15 at 17:59
  • 1
    $\begingroup$ Suppose you add a third random feature, which is not discriminative at all. That will only make things worse. Also I think another possibility is due to the perception learning algorithm, which cares only about correctly or not classifying the training data, without any measure of degree or probability. $\endgroup$ – Mou Apr 22 '15 at 18:05
  • $\begingroup$ Note that in nonlinear models (especially, say, RBF-SVMs), adding features can have much more direct negative influences. But yeah, there's really no reason to be using perceptrons rather than logistic regression / linear SVM; it's just a bad optimization algorithm for a very similar problem. $\endgroup$ – Dougal Apr 22 '15 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.