Uncertainty and confidence in classification

Question

I have a multi class classifier with 30-40 different classes. When i predict a class i would like to get an estimation about how certain my model is with its own prediction of that single sample . i.e my model predicted "Influenza" for a given sample , i would like to get a scaled number that will tell me if i want to count on its prediction for that case or not .

i'm worried that using the probability for being the class "Influenza" is not a good measure since i dont know if p=0.04 is good or bad

currently i'm using models such as Probabilistic graphical model (Bayesian network) , Boosting or logistic regression . Any ideas about a way to do that ?

Answer 1

There is a paper named "Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers". I am not deeply familiar with the article (it is in my to-read list for my own research), but its topic appears to be exactly what you need.

They presented an uncertainty estimation algorithm, which is motivated by their observation regarding the training process of deep Neural Networks using SGD. They claim that during training with SGD, reliable estimates generated in early epochs are later on deformed, not unlike overfitting for classification results.

Here is the link to the paper:

Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers