I have a (long) question regarding the evaluation of object detectors. I have two separate networks, the first one detects relevant objects (-> bounding boxes) and the second one classifies the state of the detections (think of traffic lights).

Now in the first step I simply detect objects on the test images and get bounding boxes as a result. Next I compare the detected bounding boxes to the ground truths and assign the best match to each ground truth. The problem I have is, that I dont understand how to then include the classifier into the evaluation. I think I can do it for each class separately, i.e. for each ground truth of class 'green traffic light' I will use the detected bounding box as input for the classifier. If it classifies 'green' and the IOU score from the detection is above the threshold its a true positive, if its less than the threshold its a false positive and if a ground truth is not detected at all its a false negative. With those values I can create a confusion matrix and calculate the precision, recall and f1-score for each class separately (can anyone confirm if this approach is correct?).

Now the problem is, how do I create the confusion matrix for all classes together? E.g. how do I handle not detected green ground truths in the confusion matrix, in the case of only looking at the green class it would simply be added to false negatives. In a classic multiclass image classification every ground truth (image) would be classified, either correctly or wrong, but in my case some ground truths might not get detected at all. This is how the confusion matrix should look like, where do I put a not detected ground truth (e.g. if a green traffic light was not detected)?

         green  red  yellow  off

green    [247,  82,   81,    9]

red      [45,  190,   79,    2]

yellow   [17,   81,  172,   10]

off      [0,     5,    5,   18]

Hope you can understand my problem! Thanks for any help!


1 Answer 1


You can have an extra column(to represent objects in Ground truth that were not detected) and an extra row(to represent objects that were detected but were not present in Ground Truth). The extra row could be empty(0s) in your case.

Your problem seems to be similar to Multi Object Detection, For more info:

Have a look at the 'Summary' in THIS SO answer. And THIS script computing Confusion Matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.