0
$\begingroup$

An over or under-parameterized binary classification model (- vs +) tends to over or under-fit (bias-variance tradeoff). This leads to errors during prediction on unseen data. Depending on if incorrect predictions are made on examples from class "-" or class "+", we can say something about the sensitivity and specificity of the model, with some models scoring better than others. To me, this is the link between the bias-variance tradeoff and sensitivity-specificity tradeoff for the binary classification problem [*].

What is the link between the bias-variance tradeoff and the sensitivity-specificity tradeoff for novelty detection/anomaly detection/one-class classification?

Specifically, I am interested in the case where we only have access to data from class "-", but we are still evaluating the model on a test set containing both "-" and "+" samples.

It seems that the same reasoning holds for novelty detection as for classification, but we don't run the risk of overfitting to the "+" class (only to the "-" class). Should I think differently about this trade-off for novelty detection compared to classification? Does my question point to a more fundamental misunderstanding that I might have?

[*] I recognize that you could also simply change the threshold of the model to adjust the sensitivity/specificity after training, but I do not think this is an important part of my question.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ If you only have access to data from class "-" then you do not know about sensitivity at all and not much about bias or variance. All you have is specificity, which you can maximise by making all predictions negative - any other model will perform worse. $\endgroup$
    – Henry
    Commented Aug 22, 2023 at 10:12
  • $\begingroup$ @Henry Thank you for your help. We still need to evaluate the model on a test set containing both "+" and "-" samples even though we are training only on "+" (I corrected the question to make this more clear). I suppose over-fitting on "+" could then either lead to more false positives or more false negatives, but we will only be able to measure the expected false positives with a ("+") validation set. For the false negatives, is the best we can do to stick with the simplest model possible and hope for the best? $\endgroup$
    – Douw Marx
    Commented Aug 24, 2023 at 6:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.