3
$\begingroup$

I am doing simple multi class classification ML problem.

I was given train data with perfectly balanced classes. However the data I must predict is not balanced. I was able to deduct the class proportions of test data.

Is there a way to split train data into train/validation data sets so that validation data set will have class proportions arbitrary set?

To cut it short: lot's of people want to make balanced training and validation set from imbalanced data. I want the reverse: I want to make imbalanced validation set from balanced training set;

Reasoning: I want my validation set to look like test data set; I know that 2 labels out of 7 cover 90% of data in test set (while they cover only 28% in train); I want to pass the same structure to my validation set;

$\endgroup$
  • 2
    $\begingroup$ are you trying to overfit on purpose the test set of a competition to get a better score but a worst model? (which is btw a bad idea as the validation test is private and will be released at the end) $\endgroup$ – Frayal Aug 30 '19 at 9:06
  • $\begingroup$ Yes, as I do not know how private set is created. If private is created alike the public test set then I need to over fit my models to public test set proportions. The difference is huge: train data has all 7 labels equally distributed but in public test data 3 labels cover 92% of data. As I do not know how they created private data I want to publish 2 solutions: best solution on train data proportions and best solution on test data proportions. $\endgroup$ – Dmitry Petrov Aug 30 '19 at 13:15
  • $\begingroup$ I would argue that the premise of the question is BAD practice in machine learning or statistical learning in general. What's the point of building a bad predictive model with leaked info from an already labeled test data? Just to win a competition then bake artifacts should not be encouraged here. This is misleading for the community. $\endgroup$ – doubllle Apr 30 at 8:38
0
$\begingroup$

For your idea I can recommend you to do k-fold cross validation:

enter image description here https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html https://scikit-learn.org/stable/modules/cross_validation.html

this will split your data in several train/test splits so that you avoid this unbalanced dataspread.

What I would do on top is that you should exclude some data before (randomly) to have a real test set. so that in the end after your cross validation even have one data for test left, which was not included at all.

I was given train data with perfectly balanced classes. However the data I must predict is not balanced. I was able to deduct the class proportions of test data.

this will reduce the probability to avoid this inbalanced set.

Is there a way to split train data into train/validation data sets so that validation data set will have class proportions arbitrary set?

This you really should not do, you would use knowledge from your test data, which should not be used for training.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Ok, one more time - I have train set and test set to make submission. By making fake submission I got to know that test labels are hugely imbalanced while i was given perfectly balanced. So my goal is to make validation set from train data to be imbalanced in order to mimic the test data. $\endgroup$ – Dmitry Petrov Aug 30 '19 at 13:22
0
$\begingroup$

i'm not sure about the purpose of you'r taks but you can do it with

X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    stratify=TEST_PROPORTION, 
                                                    test_size=0.25)

use the argument stratify with the proportion of each class in test set

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ But how do I make it. Obviously I cannot pass to stratify y_test as I do not have it - I only know proportions. I tried to pass just proportions but it doesn't work. Maybe I can artificially create y_test but I do not know if it will work. $\endgroup$ – Dmitry Petrov Aug 30 '19 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.