1
$\begingroup$

The labeled dataset I am using is almost 80% positive examples, 20% negative examples. However, I do not know the distribution of the data fed into the classifier.

  1. In this case, does it make sense to design the train/test set with 50% positive/negative examples?

  2. If it makes sense to rebalance the train/test set, what are strategic methods for sampling? For example, if I under sample the positive class, is there a strategic way for selecting which samples to discard or is random discard okay?

$\endgroup$
  • 1
    $\begingroup$ What are the details of your goal? (E.g. maximizing the TPR or TNR for your test set?) Those might influence what you could try out. And of course: if possible, finding out if your dataset represents the real class frequency accordingly would be beneficial. $\endgroup$ – geekoverdose Jul 8 '16 at 16:45
  • $\begingroup$ not sure what you mean as the goal is to maximize both? unfortunately can't figure out the distribution. $\endgroup$ – bla Jul 8 '16 at 18:23
  • $\begingroup$ I meant to ask for the more subtile details of your goal, like a small FPR poss. being of higher importance than a small FNR, or similar. $\endgroup$ – geekoverdose Jul 8 '16 at 18:43
1
$\begingroup$

Do not use a 50-50 dataset for testing, as that would be too far from the real usage of the model and you may be unable to get information about the classifier's actual behaiviour.

However, you can perform oversampling/undersampling of the classes in the training set in order to correct for bias in the model. My approach would be:

First, I'd try creating my training and testing sets forgetting about the imbalance. If the model I get is good, then I am happy and carry on with my life.

If that does not work as well as desired, I would split my testing set as before, but now, on the training set, remove random majority class observations until reaching balance. Now we can try to fit the model and test it.

If that does not work either (undersampling the majority class may imply a loss in valuable information), have a look at the SMOTE procedure for creating new artificial observations of the minority class.

And if that does not work either, well... Sometimes life is hard! Other approaches are converting the classification problem in an anomaly detection problem, but since the imblanace is only 4:1, I don't really think that's the best way to go.

As a final thought, take into account the relative importance of the classes (ie: What would be worse, false positives or false negatives?) and pay attention to precision or recall accordingly. Do NOT use accuracy as a performance measure when working with unbalanced data.

I hope this helped

$\endgroup$
0
$\begingroup$

In short: yes, it might make sense.

Prevalence: if you have no hard information about prevalence in your real data, you can possibly still use your domain specific knowledge to check if that prevalence might be somewhat realistic (e.g. credit card fraud vs animal prediction). If you also don't have any such information, you might just want to use the prevalence you have in your dataset (it might be this way for a reason...), or just try balanced classes, or both to see for a possible impact.

Up/downsampling: are usually done randomly (see e.g. caret::upSample and caret::downSample in R). Note that if you apply such, you usally only want to apply it to the data you use for training models to change how they fit your data - but not to any held-back test sets. Having said that, if you know your test set prevalence to be unrealistic, you might want to make it more realistic using some downsampling.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.