Is it logical to test with large test data and small train data in machine learning?

Question

I have a data frame of [1722153 X 51] out of which my data is distributed as below value indicates classes and count indicates the number of samples for each class

aggregate(data.frame(count = no_dup$id), list(value = no_dup$id), length)
  value   count
1     0 1709912 (control)
2     1    5064 (cases)
3     2    1037 (cases)
4     3    5644 (cases)
5     4     496 (cases)

I am taking a subset of the class label 0 for training the machine learning model to avoid the data imbalance issue. Below the total size, I am considering to train the machine learning model.

aggregate(data.frame(count = New_data$OVERBILL), list(value = New_data$OVERBILL), length)
  value count
1     0  4913 (control)
2     1  5064 (cases)
3     2  1037 (cases)
4     3  5644 (cases)
5     4   496 (cases)

But Can I use the remaining samples of the class 0 to test the model? Will my model face any issue wrt to classification?

Is there any possible approach that can be implemented so that I can test the remaining records?

Answer 1

I usually advice against artificially balancing data-sets, but in this extreme case you might have to, depending on the algorithm you use.

The issue is not that you have too few cases records. A couple of hundred to a couple of thousand per class should be enough for any algorithm to learn that class. No need to oversample the cases or collect more... The problem is that with those $>1000/1$ imbalances, if your controls somewhat overlap with the cases in the feature space, they might drown them out. Even in areas of low density for the controls and high density for the cases, there might still be more controls than cases because there are just so many more controls than cases in general. Most algorithms wouldn't know that the cases are much more important, for them a record is a record and they will never classify cases as cases.

I would still not balance it so completely in the training set though. I would keep 50-100k control records to let the algorithm learn with a fair amount of imbalance so that it can better deal with imbalance in the test set.

That test set should always reflect the imbalance as in real life, literally randomly sample it out of your data-set (perhaps 10-20% of total records) without stratification. Don't touch it before you have done all your model selection, feature engineering and parameter tuning within the training set.

This number of 50-100k control records kept in the training set is only a hunch of mine. You may see it as a hyperparameter to tune through cross validation within the training set.

If you take uniformly 20% of records away for a test set, that leaves 1.4M control records for training. If you decide to use "only" 100k of them, you are discarding 1.3M. Whether you should reuse them at the very end as test-set records depends on the following question: Is it relevant for you to correctly identify all controls as such? In other words, do you know the relative misclassification costs in all directions? This should tell you whether it is a waste of time/worthwhile/imperative to test all the control cases you put aside.

Those records have technically been in the training set, not the test set that you put aside at the beginning. But if you never touched them for any feature engineering, model selection or parameter tuning, you can legitimately treat them as test set records. Be aware that your test set already reflects the extreme imbalances in your data-set. If you re-add all the superfluous control records from the training set to it, you will make it even more imbalanced.

With over a million records, you also need to worry about computational complexity. For some models like random forests, complexity at test time can be high as well. Some implementations will naively only allow you to have the entire test set in memory at once, which could limit you.

Always give it a try to learn on the imbalanced data-set as is. Maybe the controls and cases don't overlap. In that case

• the imbalance doesn't create a problem identifying cases as cases.
• the imbalance doesn't affect how difficult or easy it is to tell different cases apart.
• the huge amount of controls will create a very dense point cloud for the control class which might help better identifying the controls as controls.
• you will need a good computer, but only a very simple experimental setup.

If the controls overlap with the cases though, you should do what was described above.

Answer 2

I understand what you are asking, you might want to check the following links:

• What is the difference between test set and validation set?
• Why only three partitions? (training, validation, test)
• https://www.quora.com/What-is-a-training-data-set-test-data-set-in-machine-learning-What-are-the-rules-for-selecting-them
• https://stackoverflow.com/questions/31771627/splitting-training-and-test-data?rq=1
• https://stackoverflow.com/questions/31238079/how-to-split-test-and-train-data-such-that-there-is-garenteed-at-least-one-of-ea?rq=1

Hope that helps.

Answer 3

The small number of non-control cases will be a problem for classification (i.e., you have ~140 control cases for every non-control case). You could attempt to pool all non-control cases into the one class in attempt to balance the distribution, but you will likely have more luck with an anomaly detection method considering the rarity of non-control cases.