I am using the normalized confusion matrix to aid in quantifying the uncertainty in related observations over time. More specifically, I have a classifier that returns the confidence of each class via the softmax function. I can build a confusion matrix by taking the arg max of the predictions. When I see an event, I use my classifier to produce a confidence distribution over the various classes. I then use the arg max to get the most probable class and then use the column associated with that class of the normalized confusion matrix to get the likelihood that the object is actually seen. This follows the method outlined in Tracking with Classification-Aided Multiframe Data Association (2005, Bar-Shalom).
Is there a standard way to consider the confidence of the predictions, both when building the confusion matrix and when updating the uncertainty?
One thought I had was to add the probabilities instead of the threshold value when building the confusion matrix.
The standard way of doing it
Build the normalized confusion matrix you make a matrix size CxC, where C is the number of classes. For each prediction, you take the arg max and add a vector with all zeros except for the entry from the arg max set to 1 to the column of the confusion matrix that corresponds to the correct class. You then normalize the matrix by normalizing all columns to equal 1.
The update step takes the arg max of the predictions and take that column of the confusion matrix as the prediction probability.
Proposed way
When building the confusion matrix, add the confidence vector to the column in the confusion matrix instead of the vector with a single non-zero value. When making a prediction, multiply the confusion matrix by the confidence vector to get the probability of detection.
