Perfect sampling from a huge dataset

Question

I am working with a binary predictive model for data that belongs to A and B. The learning sample that I am using contains 6000 row that belongs to group A and 1000 row that belongs to group B. I would like to make my learning sample equal in number for both variables (i.e. 1000 row that belongs to A and 1000 row that belongs to B). A random sampling technique might be very biased when selecting 1000 out of 6000 rows. What would be the best way to pick these 1000 samples from group A in a way that assures this is not a biased sampling?

In other words, I would like to have a sample of 1000 rows that represent as closely as possible the 6000 rows. What technique would do this best?

Answer 1

I will start with two (to my best knowledge) better ways of approaching the problem then subsampling the bigger set:

• Is it possible to oversample the second class instead? This way you will not lose any information (at the cost of additional computational complexity),
• Isn't your model able to use imbalanced datasets? Models like Support Vector Machines, Neural Networks etc. can deal with lack of balance on the model's definition/training algorithm execution.

If none of above is possible, answer to your question should be rather data specific. I do not think that there can exist any general way of subsampling which "represent as closely as possible", because concept of "closeness" is to broad. Depending on your application you could expect to find a subset that is close in the sense of for example:

• distribution of points (shape of the distribution)
• maximum coverage (in the sense of facility location problem, which is $NP$-hard at its own)
• minimum summarized distances (in some metric) from the subset to the set

You should provide much more details regarding your problem and used model for more detailed insight. But the choice will always be biased in some sense. The only generic approach that I can think of is performing cross-validation testing (possibly repeated many times) and selecting the best working part for some quality measure that you are concerned about (accuracy? f1? MCC?). This will also be biased, but biased towards quality of your classifier, which is your aim.

Answer 2

What classifier are you working with? I would definetly try out a random forest model for your particular problem. It is an ensemble method that consists of a large amount of orthogonal trees. This is achieved by taking a bootstrap sample of your data and then using this to grow a clasification tree. This is done a predefined number of times and then the prediction is then the aggregation of the individual trees predictions, in this case it could be a majority vote.This procedure is called bagging (Breiman 1994). Furthermore the candidate variable for each split of each tree is taken from a random sample of all the available independent variables. This introduces even more variability and makes the trees more diverse. This procedure is called the random subspace method (Ho, 1998). If you were to fit a random forest classifier you can explicitly choose the size of the sample taken from each class to grow each of the many trees. So you could grow each tree from an independant sample with balanced classes. This can work extremely well for your kind of problem as the class diffrerence is not ovewhelming. An example of how to fit this model in R is as follows (supposing you have 3 independant variables):

binaryclassifier <- randomForest(Y~X1+X2+X3,data=yourdata,ntrees=2000,
                                 sampsize=c(500,500)) # so you would end up with
                                                      # each tree grown from a 1000
                                                      # observation, balanced data set

Another thing that can come in handy here is that randomForest calculates an error measure on the fly. In this example, each tree is built on a 1000 observation bootstrap sample, meaning you left out 6000 observations. These 6000 observations are referred to as the out-of-bag sample and can be used to validate the model as the forest is progressively grown. You can finally even calculate the confusion matrix for the model based on this procedure:

binaryclassifier$confusion

You could play with sampsize a bit to, maybe, obtain an OOB accuracy you are happy with. Note that the sampsize doesnt have to be perfectly balanced for this to happen.