1
$\begingroup$

I built a machine learning two-class classification model on an approximately balanced training set and estimated its performance via cross-validation. Its accuracy is about 70%.

My boss asked me to assess the model performance on a new dataset. The new data is unbalanced: only 10% belongs to the positive class. Unsurprisingly, my model performed poorly on this data. For this data, just assigning every case to the negative class will give me 90% accuracy. My boss asked me why my model perform so poorly.

I am unable to explain the result well. Can anyone help me explain this? Especially to an audience of business folks.

$\endgroup$
  • $\begingroup$ This needs more context, which model are you using? how many data points are in the original set? how many features are in your data? $\endgroup$ – knk May 16 '18 at 15:29
1
$\begingroup$

You may put it this way.

Your model has been trained over a dataset where 50% of the population is sick, and 50% is healthy. So it has been optimized to separate "roughly" sick people from healthy people with an accuracy of 70%. The dataset was balanced, so the proportion of both groups was important. A"dummy" model which would have scored 50% (saying that everybody is healthy), so you made a significant improvement (going up to 70%).

The problem is that the population he gave you has nothing to do with what the model has learnt. Here, 90% of the population is healthy, and you have to track down "rare" sick people, which is not what the model is used to. A dummy model would have scored 90%, because it it easy to say that someone is healthy in that case (fewer chances to be wrong). In fact, in this case, the model should have been trained with another optimization metric, where the real difficulty is to label correctly sick people, because most people are healthy.

Adapt the explanation to your business (accidents, people buying on the internet, whatever...), and eventually, add a small metaphor. Something like "You are asking me to train an athlete to biathlon, and he needs to be equally good at skying and rifling. Great, I've spent half of my time on both. But then comes the competition, and you are telling me that the judges will not bother looking at its rifling performance. Had I known, I would have focused on skying instead !"

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.