I was reading a very insightful paper from booking.com published at KDD 2019 150 successful Machine Learning models: 6 lessons learned at Booking.com. The paper shares practical lessons from deploying ML models.

One interesting part is how they use the distribution of the deployed model's predictions to diagnose the model. Specifically, the followings are several different examples of the distributions over the deployed model's scores (which is from 0 to 1).

enter image description here

The paper says

  • the first case is possibly due to the high bias or high Bayes (error) in the model
  • the second case is possibly due to wrong scaling in a feature or false outliers
  • the third case (non-smooth curve) is possibly due to the deploying being too sparse
  • the fourth case is an ideal case where there is bimodal at 0 and 1

The paper doesn't provide the details of the diagnosis. Can somebody elaborate on these?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.