0
$\begingroup$

I've got an imbalanced data set on which I'm training an SVM using cross validation. I'd like to find the optimal class probability threshold that maximizes the F measure.

I've tried doing this by using the class probabilities that are found during cross validation. I've first calibrated these probabilities by training a regression model on them and then using it to find the real probabilities (Platt scaling). I then tried a range of threshold and chose the one that maximizes the F measure.

To test on an test sample, I used the tuned SVM to predicted the class probabilities. I again calibrated these using the previously learned regression model and used the threshold to assign classes.

However this results in a way worse prediction than with the default threshold and I have no clue why. The only thing I suspect that might cause this is the fact that I used the regression model on the same data as I trained it on, as this might cause the threshold to overfit a bit, but I think this would cause the found threshold to just be a bit less optimal and not dramatically worse.

Any thoughts on how I should handle this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Edit based on comments :

Yes, the idea of plats is to use the output of another model but as an additional feature and hoping that the logistic regression will perform similar as original but, with smoother probabilities. I have tried this and others from http://scikit-learn.org/stable/modules/generated/sklearn.calibration.CalibratedClassifierCV.html, mainly for Kaggle competitions and they don't seem to work that well on large and sparse datasets.

I feel the reason has more to do with the features are not suitable for logistic regression and so the final model performs worse than original xgboost/SVM on test dataset.

Something that has worked for me few times is to have the logistic regression trained on predictions of multiple classifiers (meta learners/stacking) but the effort has not really been worth the gain.

Correct link for R - http://danielnee.com/2014/10/calibrating-classifier-probabilties/

Improve this answer
$\endgroup$
3
  • $\begingroup$ I'm already using logistic regression and not lineair, I should have specified. Isn't the whole Idea of Platt scaling using the output of another model? Thanks for the usefull links! $\endgroup$
    – Henk
    Jul 19, 2018 at 23:36
  • $\begingroup$ Updated the answer, hope this helps. $\endgroup$
    – wololo
    Jul 20, 2018 at 3:36
  • $\begingroup$ Ah I've been training the regression on just the svm output instead of adding it as a feature, I'll definitely try doing it the correct way. Thanks for the R example explaining how to properly calibrate! $\endgroup$
    – Henk
    Jul 20, 2018 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.