I am training a model for the task of Binary classification using H2O.ai. The final output to the user is the probability of class_1.

Recently, I found that by default the training uses threshold moving, and all the test_f1 are the scores found after moving the threshold.

Now, Let's say the class_1 probability is 90%, while the threshold is set at 94%. The user will be completely sure that it's a true instance while the model actually classified it as false.

My question is how can We interpret the probability of a class in a task of binary classification while threshold moving has been applied?

Thank you


1 Answer 1


To be meaningful as probability values, you will want them to be calibrated. That is, when a prediction of $P$ is made, the event should happen about $P$ proportion of the times. Concretely, if you keep predicting an event to happen with $90\%$ probability, yet the event happens in only $60\%$ of the cases, your prediction of $0.9$ lacks calibration.

It is not a given that your predictions will be calibrated probability values.

The two links below discuss software tools and the statistical ideas implemented in them for assessing probability calibration. I do not know of an implementation besides R, but I once ported basic functionality of val.prob to Python without much difficulty, so it can be done.

Walk through rms::calibrate for logistic regression

Walk through rms::val.prob


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.