0
$\begingroup$

When splitting data for a classification problem one is advised to use stratified shuffling in case the target variable is skewed toward a certain class. Indeed, Sklearn has a function for that.

Suppose now that we are splitting the data w.r.t to a target variable T that is normally distributed. Is there any similar tool or technique that could split a the data set into train/val sets so that the mean/variance of T is preserved as much as possible?

I understand that for large enough sets that's already the case but I am interested in practical applications where splitting the data into train/tes/val sets skews the mean and variance by a lot.

$\endgroup$
2
  • $\begingroup$ You could just sort the data by the target variable, and then assign every $n$th data to validation set, I think? $\endgroup$ – Eoin Jan 12 at 18:20
  • $\begingroup$ What do you mean by "stratified shuffling"? The only relevant hit when searching it is the page from the sklearn documentation. I know "stratified sampling", but in that context, I don't understand the question. Is this in context of a classification you have already performed, or is this a theoretical concern? $\endgroup$ – cherub Jan 13 at 21:06
1
+50
$\begingroup$

Stratified shuffling used in the case target variable takes discrete values. "The folds are made by preserving the percentage of samples for each class," stated in sklearn documentation. So your question is how I split my data with a continuous target variable such that distrb. is preserved.

One way to do it is bin your target variable:

  • Use sklearn KBinsDiscretizer to bin your target variable
  • Assign discrete labels to each bin
  • Use stratified shuffling, with the binned variable as the target to split your data

With this technique, you should be able to preserve mean-variance in each fold given the number of bins determined accordingly.

To determine the number of bins: https://stats.stackexchange.com/a/862/293623

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.