# Is under sampling the majority population useful to predict a rare event if I limit the probabilistic classifier over 0.85?

I want to predict customers who are likely to purchase Kid Cudi's new album in a few weeks so I can perform targeted marketing. This event hasn't happened before. But I have data very similar to this event, I have album sales for Adele last year. So my data for Adele looks like this, the target variable is "adele CDs":

╔══════════╦════════════════╦═══════════════════╦══════════════╦═══════════════════╦═══════════════╗
║ customer ║ past purchases ║ total money spent ║ customer age ║ vip member status ║ adele CDs     ║
╠══════════╬════════════════╬═══════════════════╬══════════════╬═══════════════════╬═══════════════╣
║        1 ║              2 ║               400 ║           22 ║ yes               ║ purchased     ║
║        2 ║              1 ║               134 ║           19 ║ yes               ║ none          ║
║        3 ║             13 ║              1050 ║           44 ║ no                ║ none          ║
║        4 ║              4 ║               677 ║           33 ║ no                ║ none          ║
║        5 ║              4 ║               500 ║           62 ║ no                ║ none          ║
║        6 ║              7 ║               900 ║           27 ║ no                ║ purchased     ║
║        7 ║              3 ║               345 ║           21 ║ yes               ║ none          ║
╚══════════╩════════════════╩═══════════════════╩══════════════╩═══════════════════╩═══════════════╝


This would be great, except the problem is that my data of potential customers is massive (100,000) and my CDs sold is tiny (500). Every predictive model I apply results in 100% classification of the majority class - no adele CDs, and 0% classification of purchasing adele CDs.

However, if I massively under sample the majority class and keep all "yes" purchases, I can reduce my data to 500 no customers and 500 yes customers, then I see the model predicts younger customers being more likely to buy CDs as well as other patterns.

After under sampling, I was thinking I would output a probabilistic classifier with random forest, but make it such that the output has to be over a higher threshold (say 0.85) to classify, else no sale. What would you do in this situation?

• What were the exact problems you faced when using random forests? Dec 3, 2016 at 7:16
• You should try to predict the *probability that somebody will buy (logistic regression), and then targeting those where that probability is sufficiently high, even if below 50%. There are many post about this in here ... stats.stackexchange.com/questions/131255/… stats.stackexchange.com/questions/116632/… There are many Qs in here about this, but few good answers ... Dec 3, 2016 at 16:23
• Random forests work great, but they are only working if I under sample. I am aware this topic is discussed in other questions, but I believe the topic has few answers that deeply explain the techniques involved. Dec 3, 2016 at 20:03
• If that is a problem with random forrest, then maybe try a learning technique for which it is not a problem, like logistic regression? Dec 4, 2016 at 15:30
• Thanks, what benefit would logistic regression provide given that I can use random forests to output probabilistic classifiers? Dec 5, 2016 at 16:57

If you have the relative rankings, you know (or predict) the top $$N$$ people most likely to make a purchase. If you have a certain budget to spend on advertising, you can spend it on the customers most likely to make a purchase. Harrell discusses that on his blog when he refers to a "lift curve". An advantage of using the rankings over the probabilities is that you do not have to waste resources trying to pin down the exact probabilities.
Neither the rankings nor the full probability predictions are heavily affected by this kind of class imbalance. Consequently, discarding precious data is, probably, unwise. Your result of no one being predicted to buy an Adele CD comes from the fact that, under the hood, your model is applying a threshold, likely requiring a predicted probability of at least $$0.5$$ to be classified as making a purchase. When you model the probability of purchase (or at least the relative rankings), there is no such threshold.