I have an issue which I would like to seek some feedback on:

In conventional machine learning we attempt to maximise the accuracy and other metrics of our models, and are generally satisfied with over 80% accuracy.


However, what if the requirement is instead:

For any new input, if you're absolutely certain of predicting it correctly, then predict. Else, fallback to conventional systems

Intuitively, one would use the probability scores as a way of 'confidence' of the prediction. But what if the probability scores are wrong? eg, predicted a 99% probability of label A, but real answer is label B.

Is there a way to tackle such a problem? I suspect there is something wrong with the data, but we'll never have perfect data, so is there a way to solve this with the existing data?


I have tried calibrating the probabilities, but they do not seem to help much. I obtain the following calibration reliability curve: enter image description here

I have also tried ensembling, but again, it doesn't solve the problem at hand as all the classifiers are certain when predicting the wrong answer.

This has really stumped me the past few days. Would appreciate your insights on this :)

  • 2
    $\begingroup$ "if you're absolutely certain of predicting it correctly" Can you elaborate on how would you be absolutely certain of predicting something correctly? $\endgroup$
    – Ami Tavory
    Jul 12, 2017 at 8:06
  • $\begingroup$ @AmiTavory Thanks for your reply, that's my question isn't it? I only want the classifiers to predict if they are absolutely certain of predicting it correctly $\endgroup$
    – Wboy
    Jul 12, 2017 at 8:09
  • $\begingroup$ I'm probably missing something, but this sounds impossible, or at least undefined. Either your classifier is based on past patterns, in which case there's always the chance a future pattern (or noise) will be different. Alternatively, there's some deterministic rule you're allowed to use, in which case, you should find how to formulate it. $\endgroup$
    – Ami Tavory
    Jul 12, 2017 at 8:13
  • $\begingroup$ There seems to be some very strong predictors in your models, with high spikes at near 0 and 100% predicted probability. There might even be 'complete seperation' due to specific predictor variables (see stats.stackexchange.com/questions/11109/…). In other words, you might need to rethink your models, instead of some kind of recalibration based on 'absolute certainty of correctness'. Further, without using the (unknown in new cases) observed outcome I don't see a way absolute certainty would be achievable. $\endgroup$
    – IWS
    Jul 12, 2017 at 8:20
  • $\begingroup$ Hey @IWS, thanks for your reply! Could you explain more about 'rethinking your models'? What should I be doing? $\endgroup$
    – Wboy
    Jul 12, 2017 at 9:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.