# How to make train/test split with given class weights

I am doing simple multi class classification ML problem.

I was given train data with perfectly balanced classes. However the data I must predict is not balanced. I was able to deduct the class proportions of test data.

Is there a way to split train data into train/validation data sets so that validation data set will have class proportions arbitrary set?

To cut it short: lot's of people want to make balanced training and validation set from imbalanced data. I want the reverse: I want to make imbalanced validation set from balanced training set;

Reasoning: I want my validation set to look like test data set; I know that 2 labels out of 7 cover 90% of data in test set (while they cover only 28% in train); I want to pass the same structure to my validation set;

• are you trying to overfit on purpose the test set of a competition to get a better score but a worst model? (which is btw a bad idea as the validation test is private and will be released at the end) – Frayal Aug 30 '19 at 9:06
• Yes, as I do not know how private set is created. If private is created alike the public test set then I need to over fit my models to public test set proportions. The difference is huge: train data has all 7 labels equally distributed but in public test data 3 labels cover 92% of data. As I do not know how they created private data I want to publish 2 solutions: best solution on train data proportions and best solution on test data proportions. – Dmitry Petrov Aug 30 '19 at 13:15

For your idea I can recommend you to do k-fold cross validation:

this will split your data in several train/test splits so that you avoid this unbalanced dataspread.

What I would do on top is that you should exclude some data before (randomly) to have a real test set. so that in the end after your cross validation even have one data for test left, which was not included at all.

I was given train data with perfectly balanced classes. However the data I must predict is not balanced. I was able to deduct the class proportions of test data.

this will reduce the probability to avoid this inbalanced set.

Is there a way to split train data into train/validation data sets so that validation data set will have class proportions arbitrary set?

This you really should not do, you would use knowledge from your test data, which should not be used for training.

• Ok, one more time - I have train set and test set to make submission. By making fake submission I got to know that test labels are hugely imbalanced while i was given perfectly balanced. So my goal is to make validation set from train data to be imbalanced in order to mimic the test data. – Dmitry Petrov Aug 30 '19 at 13:22

i'm not sure about the purpose of you'r taks but you can do it with

X_train, X_test, y_train, y_test = train_test_split(X, y,
stratify=TEST_PROPORTION,
test_size=0.25)


use the argument stratify with the proportion of each class in test set

• But how do I make it. Obviously I cannot pass to stratify y_test as I do not have it - I only know proportions. I tried to pass just proportions but it doesn't work. Maybe I can artificially create y_test but I do not know if it will work. – Dmitry Petrov Aug 30 '19 at 13:17