I recently tried to build a classification model that tries to predict product returns based on retail data. I decided to build a random forest classifier based on features derived from both product and customer data.

When I tried to assess the performance of the classifier, I looked at the confusion matrix based on all available data points. Everything seemed to be working reasonably well; both precision and recall as well as accuracy had acceptable values.

I furthermore decided to look at the classifier's performance separated by different properties, such as customer location (city, province, etc.), age, and so forth.

To my surprise, I found that the resulting confusion matrices (which are conditional based on the separating properties) exhibited a large degree of variability. Matrices for some properties had very high numbers of false positives and low numbers of false negatives, while others had the opposite error distribution. The fact that the confusion matrix for all data points looks OK seems to be due to a "balance of errors" of some sorts.

I recently came across this article which discusses a similar issue in the context of predicting criminal behaviour:


Any help or insights would be greatly appreciated!

  • $\begingroup$ Your question makes me think that Feature Selection could give you some insights. Depending on the data (and the extracted features), it can be important step when training the classifier that can drastically change its performance. I would be interested to see the results you obtain without these irrelevant features (if some are). Things should globally be more stable. $\endgroup$ – Eskapp Nov 11 '16 at 17:29
  • 1
    $\begingroup$ Thanks for your suggestion. I tried a number of different combinations of feature subsets based on feature importances estimated by the random forest classifier and the problem persists... $\endgroup$ – Martin Keller Nov 17 '16 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.