I've got an imbalanced data set on which I'm training an SVM using cross validation. I'd like to find the optimal class probability threshold that maximizes the F measure.
I've tried doing this by using the class probabilities that are found during cross validation. I've first calibrated these probabilities by training a regression model on them and then using it to find the real probabilities (Platt scaling). I then tried a range of threshold and chose the one that maximizes the F measure.
To test on an test sample, I used the tuned SVM to predicted the class probabilities. I again calibrated these using the previously learned regression model and used the threshold to assign classes.
However this results in a way worse prediction than with the default threshold and I have no clue why. The only thing I suspect that might cause this is the fact that I used the regression model on the same data as I trained it on, as this might cause the threshold to overfit a bit, but I think this would cause the found threshold to just be a bit less optimal and not dramatically worse.
Any thoughts on how I should handle this?