I have a binary SVM with probability output (via Platt scaling). I want to set a threshold on the probability outputs since I want to trade off making false positives/negatives. Is it possible to interpret the threshold as (1 - false positive rate) of the resulting classifier? (Prior probability of positives is very low i.e. close to 0). I understand that one can estimate false positive and false negative rate for a given threshold with evaluating the resulting classifier on the test set. I do however have only very few samples and thus the resulting scores would be imprecise. Making me wonder if I could get a better estimate directly out of the threshold value. After all changing the threshold should have an effect on the false positive and false negative rate even if it doesn't lead to different scores (due to small data set).
Edit: After some research I found out that deciding the threshold based on the test data is a bad idea due to overly optimistic performance measures. Therefore I would have to use train/validation/test splitting where the threshold is decided with the validation data. However the question whether the threshold probability can be treated as (1-FPR) still holds. The threshold probability is probably a worse estimator of the FPR though since it is only estimated with the training data which was already used for training which might make the estimation overly optimistic.
EditEdit: Now that I think about it, the FPR estimate might not be overly optimistic due to the probability estimates being trained via a hold out set during the Platt scaling process.