I am doing a document classification challenge on hackerrank.com. The training data is $X$ strings classified into $8$ classes. My approach is to use word frequencies and naive bayes (
sklearn) instead, since I've read this is good for document classification. Specifically, I found the $Y$ most common words for each class, put this set $Y$ for each class into a large set of all common words. The features were then how many times each word in this large set occurred in the string.
The result of this for naive bayes classifier was a
sklearn.metric.accuracy_score (on unseen data) of less than $0.75$ while other methods such as
sklearn.tree.DecisionTreeClassifier achieved a $0.8$ to $0.9$. These scores were for choosing $Y$ (number of words from each class to use as features) to around 100. For $Y=10, 4$ etc the score was much lower.
Considering the cutoff accuracy is $0.95$, I am wondering what I have done wrong for two reasons:
- Why are svm and decision tree performing better than naive-bayes, for a problem naive bayes is supposedly good at?
- Is there something fundamentally wrong with my feature selection or choice of methods preventing me from achieving a higher score, or is it likely due to some error in the code?
Note that I have tried other features, like avg. word length, number of characters, with worse results for all methods.