1
$\begingroup$

I am using perceptron to solve a classification problem. I have a limited amount of features (26) and iterate through all possible combinations of them. A combination of two features [feature_a, feature_b] results in better accuracy than the same combination with additional third feature [feature_a, feature_b, feature_c]. Is there any reasonable explanation for that apart from a bug in my code?

I tried replacing perceptron by naive bayes classifiers, the pattern persists.

I am using scikit-learn Python library for perceptron and NB implementations, if that's important.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think that is the problem of overfitting. Adding features can always improve training acc. in such a shallow model, but may hurt generalization.

p.s., I think you may try logsitic regression or linear SVM rather than perception.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I thought overfitting would occur if I had much more features than 3, is it possible with such a simple model? $\endgroup$
    – MarieZ
    Commented Apr 22, 2015 at 17:59
  • 1
    $\begingroup$ Suppose you add a third random feature, which is not discriminative at all. That will only make things worse. Also I think another possibility is due to the perception learning algorithm, which cares only about correctly or not classifying the training data, without any measure of degree or probability. $\endgroup$
    – Mou
    Commented Apr 22, 2015 at 18:05
  • $\begingroup$ Note that in nonlinear models (especially, say, RBF-SVMs), adding features can have much more direct negative influences. But yeah, there's really no reason to be using perceptrons rather than logistic regression / linear SVM; it's just a bad optimization algorithm for a very similar problem. $\endgroup$
    – Danica
    Commented Apr 22, 2015 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.