0
$\begingroup$

I'm primarily looking to attain the maximum True Positive Rate for a small False positive Rate of say 0.01. The following is an instance:

1, 37.33, 228.39, 0, 77.060599, 0.073384, 0.052536, 1.389826, 0.526793, -0.806316, 20.302738, 6

There are 100k instances of each class available. I understand that a thorough experimentation can be required, but what would be your first set of classifiers to attack this classification problem?

I have tried the default scikit-learn implementations of Random Forests, Gradient Boosted Classifier and Stochastic Gradient Descent with logistic loss function. As of now RF seems to be leading. But is it possible for SGD to beat RF at larger datasets? Or is there anything else that has any real chance of beating RF in this case?

$\endgroup$
1
$\begingroup$

RF is very robust and powerful, but it can succumb to over-fitting by having small trees for each case it is exposed to and fail miserably on unseen data. Which method works better really depends on your data. A couple things look at:

  1. Make sure you are evaluating performance on data that the model has never seen.
  2. Check the feature importances you get out of RF. You might find some features are useless and may be causing issues with SGD.
  3. SGDClassifier has many different options, so a parameter sweep might reveal a better configuration.
  4. SGD is a linear model really, so perhaps you could try one of the Naive Bayes models or a Neural Network model.
$\endgroup$
1
$\begingroup$

Because of the no free lunch theorems, there is no definite answer to your question. It depends on your data, more specifically how its structure happens to align with the implicit assumptions made by various learning methods. In reality, the only way to know which method works best is to try them all (and properly optimize hyperparameters of every method to ensure a fair comparison!).

That said, you are primarily interested in a high precision model. Most general purpose methods (RF, neural networks, SVM, ...) allow you to specify misclassification costs per class in some way. To get a high precision classifier, use a very large penalty on misclassification of negatives and you will get a very conservative model (= that is one that yields very few positives, but with high precision).

Finally, make sure you inspect ROC curves to decide on an appropriate cutoff per model, rather than basing your evaluation on the (somewhat arbitrary) default cutoff that is used within your models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.