Suppose we have a machine-learning model already in use in the context of credit card application approval process: the application will be successful if the model predicts that the applicant is a good customer, and the application will fail if the applicant is predicted to be a bad customer or even a fraud.

The model has been running for several years, and has a decent performance (within the customers who own our credit card, only 5% of them are bad customers). now it is time to improve our existing models, the question is how?

we have all the data of all the applicants regardless of their applications being successful or not. following are the several options plus questions:

  1. we consider all the failed customers whose application didn't pass through as bad customers, and we consider those 5% bad customers as, obviously, bad customers. Then I train a binary classification model. the model might gives us a lower percentage of bad customers (lower false negative number), with the price of much higher false positive number, which means less people will be accepted as potential customer, and the model might turn out to be completely useless. how to fix that?

  2. the ideal model will gave us higher acceptation rate and lower bad customer rate, which is almost impossible from a model. the second would be higher acceptation rate and a slightly higher bad customer rate, how to use our data to reach this goal?

  3. are there other methods to improve the existing model?

  • 1
    $\begingroup$ I think this is a variation of the exploration/exploitation trade-off problem, which I know from reinforcement learning. How it plays out when a supervised learning model is used to perform interventions, and your classification problem may be non-stationary, I don't know. But I think it is a very interesting question and very important as companies look toward 2nd and 3rd-generation ML models embedded as decision assistants. $\endgroup$ – Neil Slater Aug 31 '17 at 8:36
  • $\begingroup$ You should not look at this as a classification problem, better as risk estimation (logistic regression). Error rate is not a proper score function, search this site. $\endgroup$ – kjetil b halvorsen Dec 9 '18 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.