SVM probability output threshold as (1 - FPR)?

Question

I have a binary SVM with probability output (via Platt scaling). I want to set a threshold on the probability outputs since I want to trade off making false positives/negatives. Is it possible to interpret the threshold as (1 - false positive rate) of the resulting classifier? (Prior probability of positives is very low i.e. close to 0). I understand that one can estimate false positive and false negative rate for a given threshold with evaluating the resulting classifier on the test set. I do however have only very few samples and thus the resulting scores would be imprecise. Making me wonder if I could get a better estimate directly out of the threshold value. After all changing the threshold should have an effect on the false positive and false negative rate even if it doesn't lead to different scores (due to small data set).

Edit: After some research I found out that deciding the threshold based on the test data is a bad idea due to overly optimistic performance measures. Therefore I would have to use train/validation/test splitting where the threshold is decided with the validation data. However the question whether the threshold probability can be treated as (1-FPR) still holds. The threshold probability is probably a worse estimator of the FPR though since it is only estimated with the training data which was already used for training which might make the estimation overly optimistic.

EditEdit: Now that I think about it, the FPR estimate might not be overly optimistic due to the probability estimates being trained via a hold out set during the Platt scaling process.

Answer 1

assuming that your probability predictions are well-calibrated, for p=0.1 you should expect 10% of data points to be in the positive group and 90% in the negative group regardless of the balance of the classes. That means that this threshold is not related to TPR/FPR but PPV and NPV. Since it's telling you given predictions, what is the probability the data point is positive (PPV) and not given that the data point is positive, what's the probability that I will get positive prediction (TPR)

Although the plat scaling is fitted in CV, that doesn't mean that if you look at the probabilities in your training set these will not be biased, they probably will. For example, all support vectors from one group are by definition on the same hyperplane and they will get the same predicted probability