1
$\begingroup$

I have a classification problem with unbalanced classes where the natural proportion of the positive class is 0.02% but the total number of cases is high (+300k) which allows to change it when training the model using a higher rate (+2% of positive class).

I am doing a CrossValidation with 10 folds keeping that rate = 2% and then I have a test set that has a natural rate of 0.02%.

What I would like to know is: Should the validation set in the K-folds have the natural rate (0.02%) or the it's ok to have a 2% rate since we are testing later on a 0.02% test-set ?

Improve this question
$\endgroup$
1
  • $\begingroup$ Stratified sampling would be a better choice. As for controlling the rate to match the test set... that will be problematic, better to use a metric which doesn't rely on this ratio. $\endgroup$
    – user2974951
    Sep 24, 2020 at 8:09

1 Answer 1

Reset to default
0
$\begingroup$

Usually, it's better if validation mimics the test set. This is to ensure your validation score is close to your test score. Because, you're optimising validation performance in CV loop, with the motivation of performing well on the test set.

Here, you have the test set, so you can make it happen but sometimes, this might not be even possible. For example, assume you train a classifier to detect lung cancer. The dataset you'll have might not represent the cancer ratio across the entire population (the cancer ratio in the population can be much much smaller). And, when you use it as a diagnostic tool, you may encounter a much different test set. On the other hand, it there was such a diagnostic tool, it'd keep on learning with new data.

So, this is not a necessity (e.g. no data leakage); you can manipulate the data as you want leaving the test set untouched, but it's your own risk. I'd usually expect to find a better a model when distributions match.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is a bad idea, as soon as the test set changes, and so the ratio in the test set changes, it won't work well anymore. You should not modify your training based on your test set (characteristics), it is still data leakage. Better to use a metric which does not depend on the ratio of classes, and not for ex. AUC, which is highly dependent on this ratio. $\endgroup$
    – user2974951
    Sep 24, 2020 at 8:53
  • $\begingroup$ Actually, I need to explain myself a bit more I guess. If you're given a test set, and your training set is not from the same distribution, there is little to do, because as you said we leak the ratio information in the test set. But, if we're splitting the entire set as train/test ourselves, they should be similar, which is why we use stratified sampling (in validation as well). Or, if we have a prior information that the natural positive ratio is %0.02 (w/o actually looking into the test set), we should reflect this into training and validation steps. $\endgroup$
    – gunes
    Sep 24, 2020 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.