Does anyone know if the following has been described and (either way) if it sounds like a plausible method for learning a predictive model with a very unbalanced target variable?
Often in CRM applications of data mining, we will seek a model where the positive event (success) is very rare relative to the majority (negative class). For example, I may have 500,000 instances where only 0.1% are of the positive class of interest (e.g. the customer bought). So, in order to create a predictive model, one method is to sample the data whereby you keep all positive class instances and only a sample of the negative class instances so that the ratio of positive to negative class is closer to 1 (maybe 25% to 75% positive to negative). Over sampling, undersampling, SMOTE etc are all methods in the literature.
What I am curious about is combining the basic sampling strategy above but with bagging of the negative class.Something simply like:
- Keep all positive class instances (e.g. 1,000)
- Sample the negative classe instances in order to create a balanced sample (e.g. 1,000).
- Fit the model
- Repeat
Anyone hear of doing this before? The trouble it seems without bagging is that sampling only 1,000 instances of the negative class when there are 500,000 is that the predictor space will be sparse and you may well not have a representation of possible predictor values/patterns. Bagging seems to help this.
I looked at rpart and nothing "breaks" when one of the samples does not have all the values for a predictor (does not break when then predicting instances with those predictor values:
library(rpart)
tree<-rpart(skips ~ PadType,data=solder[solder$PadType !='D6',], method="anova")
predict(tree,newdata=subset(solder,PadType =='D6'))
Any thoughts?
UPDATE: I took a real world data set (marketing direct mail response data) and randomly partitioned it into training and validation. There are 618 predictors and 1 binary target (very rare).
Training:
Total Cases: 167,923
Cases with Y=1: 521
Validation:
Total Cases: 141,755
Cases with Y=1: 410
I took all positive examples (521) from the training set and a random sample of negative examples of the same size for a balanced sample. I fit an rpart tree:
models[[length(models)+1]]<-rpart(Y~.,data=trainSample,method="class")
I repeated this process 100 times. Then predicted the probability of Y=1 on the cases of the validation sample for each of these 100 models. I simply averaged the 100 probabilities for a final estimate. I deciled the probabilities on the validation set and in each decile calculated the percentage of cases where Y=1 (the traditional method for estimating the ranking ability of the model).
Result$decile<-as.numeric(cut(Result[,"Score"],breaks=10,labels=1:10))
Here is the performance:
To see how this compared to no bagging, I predicted the validation sample with the first sample only (all positive cases and a random sample of the same size). Clearly, the data sampled was too sparse or overfit to be effective on the hold out validation sample.
Suggesting the efficacy of the bagging routine when there is a rare event and large n and p.