3
$\begingroup$

I have a multi class classifier with 30-40 different classes. When i predict a class i would like to get an estimation about how certain my model is with its own prediction of that single sample . i.e my model predicted "Influenza" for a given sample , i would like to get a scaled number that will tell me if i want to count on its prediction for that case or not .

i'm worried that using the probability for being the class "Influenza" is not a good measure since i dont know if p=0.04 is good or bad

currently i'm using models such as Probabilistic graphical model (Bayesian network) , Boosting or logistic regression . Any ideas about a way to do that ?

Improve this question
$\endgroup$
3
  • $\begingroup$ What about using the odds ratio from logistic regression? $\endgroup$ – user2974951 Jan 31 '19 at 10:35
  • $\begingroup$ Or repeating the prediction with many bootstrapped samples. This will work for any classifier. $\endgroup$ – peteR Jan 31 '19 at 10:42
  • $\begingroup$ @peteR Can you please elaborate some more? $\endgroup$ – Latent Jan 31 '19 at 11:18
3
$\begingroup$

There is a paper named "Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers". I am not deeply familiar with the article (it is in my to-read list for my own research), but its topic appears to be exactly what you need.

They presented an uncertainty estimation algorithm, which is motivated by their observation regarding the training process of deep Neural Networks using SGD. They claim that during training with SGD, reliable estimates generated in early epochs are later on deformed, not unlike overfitting for classification results.

Here is the link to the paper:

Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.