1
$\begingroup$

According to its Wikipedia page, receiver operating curves are created by plotting the TPR vs. the FPR at various discrimination thresholds where:

TPR = TP / (TP + FN)

FPR = FP / (FP + TN)

What would be the process of plotting this ROC curve with an object detection model? Can this only be accomplished with a multi-class object detection model and not single class?

To my understanding, we must pit a class against all other classes to come up with an ROC curve (at least specifically with obj det models). That is, if we have classes A, B, and C, then an ROC curve for class A would be class A vs B and C, and so on. It is impossible to have an ROC curve otherwise because we cannot calculate TN's and FN's with object detection of a single class.

If the above is correct, would the below be reasonable to calculate the ROC curve of class A vs B, C? (Specifically, is my understanding of FP's, FN's, TP's, TN's and TPR and FPR in this context at each iteration correct?)

  1. Run inference on a test dataset, saving predictions in a list in the order of confidence scores (descending). A single prediction consists of a confidence score, bounding box, and label.

  2. Iterate through the list of predictions. Calculate FP, FN, TP, TN at each iteration where:

    a. False Positive = model predicted an object as A when the object is not A (could be B, C, or nothing) or the A bounding box does not exceed the IOU thresh.

    b. False Negative = model predicted an object as B or C but it was not B or C, or the B or C prediction was correct but the bounding box did not exceed the IOU thresh.

    c. True Positive = model correctly predicted class A and the A bounding box exceeds the IOU thresh.

    d. True Negative = model correctly predicted class B or C and the B or C bounding box exceeds the IOU thresh.

  3. With these values at each iteration, we then add them to a global count of each. For eg. global_false_positive += iter_false_positive).

  4. With each global value at each iteration, we can calculate the TPR and FPR up to that confidence score (row). This (TPR, FPR) pair is a coordinate in the ROC curve.

  5. Plot FPR vs TPR. This gives us the ROC curve for Class A vs B and C. Repeat the two for B vs A and C and C vs A and B.

$\endgroup$
0
$\begingroup$

True Negative (TN): Does not apply. It would represent a corrected misdetection. In the object detection task there are many possible bounding boxes that should not be detected within an image. Thus, TN would be all possible bounding boxes that were corrrectly not detected (so many possible boxes within an image). That's why it is not used by the metrics.

So, no TN, no FPR, no ROC curve!

Origin at https://github.com/rafaelpadilla/Object-Detection-Metrics.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Metrics like TPR, FPR, and AUC-ROC are defined for the binary classification problem. When you have more than two classes it is necessary to first transform the problem into a binary classification problem. The usual strategy for doing this is called one vs. all (although the nomenclature is not very standardized) and is exactly what you intuited it should be: treat one class as the positive class and all others as the negative class. This gets you one ROC curve per class.

There are actually two common strategies for combining such metrics into an average across all classes: micro averages and macro averages. The macro average simply computes the metric for each class, then averages them together. The micro average is closer to what you propose, and adds together the numbers of true positives and false positives before taking ratios. Arguably the micro approach is less misleading, but that's grossly simplifying. Read the linked answer for a more detailed discussion.

The strategy you outline for drawing a combined ROC curve is basically to compute the micro averaged TPR and FPR for a series of thresholds, then draw an ROC curve from that. That certainly gets you a curve but I'm not sure I would still call it an ROC curve. As the curve goes up, the model is probably doing better in some sense, although I would worry that it would obscure the trade-offs being made under the hood.

Personally, I would find such a chart difficult to interpret. I would prefer to see aggregate metrics like accuracy (which is defined for the multiclass problem.) That is how results for ImageNet are usually reported, for example. Aggregate $F_1$ scores or AUC scores computed via micro averaging might also be interesting. Too much information obscures; we want a single metric that simply represents "goodness." Such single number metrics could be used for hyper-optimization, for example.

It is usually more instructive to look at misses than abstract ROC curves. For example, when an algorithm misclassifies a "housecat", does it label it a "tiger" or a "bulldozer?" Does it fail in bad light, from unusual angles, when the image is inverted or rotated, or in black-and-white? These insights more directly lead to concrete actions that can be taken to improve the algorithm or augment the training data.

You mention image detection and metrics like Intersection-over-Union which suggest you are interested in the object localization problem - finding where exactly in the image the object is located. The ImageNet challenge defines a metric for that, which you can find on their website.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.