This question might be strange but I am really disappointed with the bad results I have been getting so I decided to get some ideas from experts. I have a dataset X of binary classes y=(1,-1). This dataset has 100,000 samples and 128 features. Each feature is basically a binary value 0 or 1. For instance, one sample would look like: x=[0,1,0,0,0,1,1, .... ,1,1], y=[-1]. Using scikit-learn, I have tried the following classifiers: Random Forests, Naive Bayes, Linear SVC, and Multi-layer Perceptron classifier. For instance, using LinearSVC():

from sklearn.model_selection import cross_val_score
from sklearn.svm import LinearSVC

clf = LinearSVC()
score = cross_val_score(clf, X, y, cv=10)
print("Accuracy= %0.2f (+/- %0.3f)" % (score.mean(), score.std() * 2))

For each one, I used Cross-Validation of 10-Folds. As a result, none of these classifiers exceeds 50% accuracy!

I am not sure what is wrong! The dataset is generated from a reliable source (from my research team). I cannot think of any other way so I am asking for hints from those who might have faced such a problem before.

Thank you in advance.

  • $\begingroup$ Is accuracy a reasonable metric for your application? How are you converting predicted probabilities into class assignments? $\endgroup$ – Matthew Drury Oct 27 '17 at 22:43
  • $\begingroup$ @MatthewDrury I added an example in my post. $\endgroup$ – steve Oct 27 '17 at 22:56
  • 1
    $\begingroup$ Don't use .score on classifiers in sklearn (like, seriously, never, it's poorly thought out). It compares to a threshold of 0.5, and that is (almost) never a reasonable thing to do. You need to tune your probability threshold based on what false positive and false negative rates are reasonable for your application. $\endgroup$ – Matthew Drury Oct 27 '17 at 23:02

There is no guarantee that a large number of records and/or a large number of variables lead to good classification accuracy.

The variables need to have predictive power for the class. For example, you would have a hard time predicting loan defaults if your input variables describe statistics about social media usage of your applicants. In this case it also wouldn't necessarily help to analyze their social media usage in more detail and/or to have more applicants in your data-set. The predictors need to be relevant to the question you want answered: You would need financial ratios etc.

That was of course a caricature of an example, but I wouldn't be so sure that input data that intuitively seems to be relevant always is. That's a separate issue from the data being reliable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.