1
$\begingroup$

I have a binary SVM with probability output (via Platt scaling). I want to set a threshold on the probability outputs since I want to trade off making false positives/negatives. Is it possible to interpret the threshold as (1 - false positive rate) of the resulting classifier? (Prior probability of positives is very low i.e. close to 0). I understand that one can estimate false positive and false negative rate for a given threshold with evaluating the resulting classifier on the test set. I do however have only very few samples and thus the resulting scores would be imprecise. Making me wonder if I could get a better estimate directly out of the threshold value. After all changing the threshold should have an effect on the false positive and false negative rate even if it doesn't lead to different scores (due to small data set).

Edit: After some research I found out that deciding the threshold based on the test data is a bad idea due to overly optimistic performance measures. Therefore I would have to use train/validation/test splitting where the threshold is decided with the validation data. However the question whether the threshold probability can be treated as (1-FPR) still holds. The threshold probability is probably a worse estimator of the FPR though since it is only estimated with the training data which was already used for training which might make the estimation overly optimistic.

EditEdit: Now that I think about it, the FPR estimate might not be overly optimistic due to the probability estimates being trained via a hold out set during the Platt scaling process.

$\endgroup$

1 Answer 1

1
$\begingroup$

assuming that your probability predictions are well-calibrated, for p=0.1 you should expect 10% of data points to be in the positive group and 90% in the negative group regardless of the balance of the classes. That means that this threshold is not related to TPR/FPR but PPV and NPV. Since it's telling you given predictions, what is the probability the data point is positive (PPV) and not given that the data point is positive, what's the probability that I will get positive prediction (TPR)

Although the plat scaling is fitted in CV, that doesn't mean that if you look at the probabilities in your training set these will not be biased, they probably will. For example, all support vectors from one group are by definition on the same hyperplane and they will get the same predicted probability

$\endgroup$
4
  • $\begingroup$ Now that I think more closely about it. I think the chosen threshold does not have anything to do with the data distribution. To cite sklearn here: "[...] a well calibrated (binary) classifier should classify the samples such that among the samples to which it gave a predict_proba value close to 0.8, approximately 80% actually belong to the positive class.". This means that the output is merely a confidence level and has nothing to do with the underlying data distributions. It's like comparing apples with oranges. $\endgroup$
    – Philipp
    Oct 30, 2019 at 16:23
  • $\begingroup$ I agree with you that the Plat scaling is introducing a bias even with the CV. The CV part is just trying to lower the introduced bias. $\endgroup$
    – Philipp
    Oct 30, 2019 at 16:34
  • $\begingroup$ The support vector example I don't understand. Is what you are saying that SVMs are inherently biased due to all support vectors of the same group getting the same value? I don't think that is a bias though since there is no necessity of having multiple support vectors per group. $\endgroup$
    – Philipp
    Oct 30, 2019 at 16:40
  • $\begingroup$ depends on your data. IT's not unusual to 90% of samples to be support vectors in some situations. That happens when the decision boundary is very complex or when you have high dimensional data. $\endgroup$
    – rep_ho
    Oct 31, 2019 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.