I have a (long) question regarding the evaluation of object detectors. I have two separate networks, the first one detects relevant objects (-> bounding boxes) and the second one classifies the state of the detections (think of traffic lights).
Now in the first step I simply detect objects on the test images and get bounding boxes as a result. Next I compare the detected bounding boxes to the ground truths and assign the best match to each ground truth. The problem I have is, that I dont understand how to then include the classifier into the evaluation. I think I can do it for each class separately, i.e. for each ground truth of class 'green traffic light' I will use the detected bounding box as input for the classifier. If it classifies 'green' and the IOU score from the detection is above the threshold its a true positive, if its less than the threshold its a false positive and if a ground truth is not detected at all its a false negative. With those values I can create a confusion matrix and calculate the precision, recall and f1-score for each class separately (can anyone confirm if this approach is correct?).
Now the problem is, how do I create the confusion matrix for all classes together? E.g. how do I handle not detected green ground truths in the confusion matrix, in the case of only looking at the green class it would simply be added to false negatives. In a classic multiclass image classification every ground truth (image) would be classified, either correctly or wrong, but in my case some ground truths might not get detected at all. This is how the confusion matrix should look like, where do I put a not detected ground truth (e.g. if a green traffic light was not detected)?
green red yellow off
green [247, 82, 81, 9]
red [45, 190, 79, 2]
yellow [17, 81, 172, 10]
off [0, 5, 5, 18]
Hope you can understand my problem! Thanks for any help!
