I've analyzed a data set from a credit card company before and it has the famous "unbalanced classes" problem like all the other credit card companies, i.e., in the data set (the information of users who have been issued the credit card), there are very small fraction of data are default (less than 5%) and the majority of the data are non-default. In the literature, there are many ways to deal with this unbalance problem, e.g., under-sampling the majority class or over-sampling the minority class.
My question is the following: (1) why unbalance data in this case is a problem at all, e.g., will this issue be a problem in evaluating new classify algorithm? or something else, what are they? I know and have seen in my analysis that this will lead to substantial within-class mis-classification rates even if the overall rate is low, but is that a problem? (2) If I want to develop a new algorithm to make the accept/reject decision for the company, how do I know my new algorithm is "better" than the current algorithm that generates the data? (without running a field experiment, e.g., from now on, use my new algorithm and run for a year to see)
Here are some of my thoughts: First, for the credit card company, the mis-classification rate (the fraction of the card issued that are default) is not the only target of the company, since if that is the only goal, then the company could set a very tight rule such that it only accept a very tiny fraction of applications and make the default rate to be 0. In this case, the company will lose much potential revenue.
I assume the data set collected from a credit card company balances the mis-classification rate and the revenue maximization goals. Now if I want to develop a new classification algorithm and try to evaluate this new algorithm comparing to the current one that generates the data, how to do that? I think here the problem of the unbalance data set comes to play: given the current data set (I mean the data set of those issued card users. I think even if you have the initial application data set, that will not be helpful, since for those applications that has been declined by the current algorithm, you cannot observe whether those customers default or not, that is just physically unobservable), it is very likely to say that if I just blindly classify everyone in the data set as "non-default", then it will still achieve about 95% successful rate. However, if you apply this "new" algorithm tomorrow, very likely in a year (or less than that), you will find a lot of card you issued has default (but whether it is good or bad from total revenue maximization perspective is another thing).
If you run random forest, Neural nets, etc., on this data set, you will probably reach the same successful rate around 95%, which is the same as the current algorithm. But I doubt that if you find your new algorithm is better than the current one (on another test data set that is also generated by the current algorithm but is not used to fit your new algorithm), then your new algorithm will have better performance if you apply it on the natural application data set, i.e., run a field experiment. Then how could we evaluate a different algorithm without running field experiment? According to the under/over sampling method, one could under sampling the majority class from the current data set and then derive a decision function, then do another bootstrap from the current data set, which also under sampling the majority class and derive another decision function. You could do it for multiple times and eventually, do bagging, i.e., average all of these decision functions, which results in your new algorithm. Then you can apply this new algorithm on a test data set, which is also generated by the current algorithm but are not used to train your new algorithm. I am not sure (1) whether the new algorithm will outperform the current one on this test data set conceptually; (2) even if the new algorithm is better (or worse) than the current one on the test data set, does it mean if I apply the new algorithm on the natural application data set, the new algorithm will be better (or worse) than the current one?