I recently tried to build a classification model that tries to predict product returns based on retail data. I decided to build a random forest classifier based on features derived from both product and customer data.
When I tried to assess the performance of the classifier, I looked at the confusion matrix based on all available data points. Everything seemed to be working reasonably well; both precision and recall as well as accuracy had acceptable values.
I furthermore decided to look at the classifier's performance separated by different properties, such as customer location (city, province, etc.), age, and so forth.
To my surprise, I found that the resulting confusion matrices (which are conditional based on the separating properties) exhibited a large degree of variability. Matrices for some properties had very high numbers of false positives and low numbers of false negatives, while others had the opposite error distribution. The fact that the confusion matrix for all data points looks OK seems to be due to a "balance of errors" of some sorts.
I recently came across this article which discusses a similar issue in the context of predicting criminal behaviour:
Any help or insights would be greatly appreciated!