Metrics for one-class classification

Question

How do you calculate precision and recall in one class classification? In other words in one class classification, we just have TP(True Positive) and FN(False Negative). Which metrics we should use for these type of classification?

Answer 1

We use one-class classification is used when we have only "positive" labels (although some argue for using it when the quality of the data about the labels is poor) for outlier, or anomaly, detection.

With such data you cannot assess accuracy of the predictions. Technically you can check if it properly labeled all your data as "positive", but then you would conclude that the useless model that always returns "positive" label no matter of data, has perfect fit.

To judge performance of such classifier you would need to have data with "negative" labels. One thing you could do is to simulate data with artificially introduced anomalies (this is often done, e.g. in image classification where you add noise to the data, or transform the images), or simulate such data that you know that should be classified as anomaly, and use such data for testing.

The story is different if you have data about "positive" and "negative" classes, since then you can use exactly the same tools for evaluating your model as for classification in general, but then, why would you use one-class classification algorithms?

Answer 2

Though it's a late reply, I'd like to point of implicit assumptions by previous answers that likely don't hold.

1. for one-class classification, we don't know the real ratio of positive and negative data. So we cannot any development set has similar distribution to the real data.
2. A standard setting for one-class classification is we have positive and unlabeled dataset. We can't assume we have the label for "negative" data even in the development set. Also, we can't assume all the unlabelled data are "negative".

An alternative evaluation is proposed in the following paper (section 4):

Lee, Wee Sun, and Bing Liu. "Learning with positive and unlabelled examples using weighted logistic regression." ICML. Vol. 3. 2003.

They uses

$$ \frac{r^2}{\Pr(Y=1)} $$

P.S.: Prof. Lee, Prof Liu and Dr. Cheng are the people that coined one-class classification. We can take their evaluation as somewhat "official".

Answer 3

@user3791422 has the right answer. In addition, I would like to point out:

If you have the notion of True Positive and False Negative, it means you have a notion of the ground truth and you have predicted responses. Therefore, by definition, False Positive and True Negative should exist.

Logically, what the OP didn't consider is, if we know some examples belong in the class we know the remaining examples (rest of the world population for which we don't actually need training examples) belong outside the class. Therefore, FP and TN can be calculated when we observe them during testing.

Answer 4

When you do one class classification, besides TP and FN, you should also have FP (false positive) and TN (true negative). FP are the instances that you have classified as positives when they were actually negative and TN are the instances that you have correctly classified as negatives. Then you can calculate precision and recall:

precision = TP/(TP + FP) recall = TP/(TP + FN)

The wikipedia page https://en.wikipedia.org/wiki/Precision_and_recall explains it very well.

However, when you do one class classification, some other common metrics are the false positive rate (FPR) and the f1-score.

FPR = FP/(FP+TN)

F1-SCORE = 2TP/(2TP+FP+FN)

I hope that helped. Regards!