I have a multi class classifier with 30-40 different classes. When i predict a class i would like to get an estimation about how certain my model is with its own prediction of that single sample . i.e my model predicted "Influenza" for a given sample , i would like to get a scaled number that will tell me if i want to count on its prediction for that case or not .
i'm worried that using the probability for being the class "Influenza" is not a good measure since i dont know if p=0.04 is good or bad
currently i'm using models such as Probabilistic graphical model (Bayesian network) , Boosting or logistic regression . Any ideas about a way to do that ?