0
$\begingroup$

When splitting data for a classification problem one is advised to use stratified shuffling in case the target variable is skewed toward a certain class. Indeed, Sklearn has a function for that.

Suppose now that we are splitting the data w.r.t to a target variable T that is normally distributed. Is there any similar tool or technique that could split a the data set into train/val sets so that the mean/variance of T is preserved as much as possible?

I understand that for large enough sets that's already the case but I am interested in practical applications where splitting the data into train/tes/val sets skews the mean and variance by a lot.

Improve this question
$\endgroup$
2
  • $\begingroup$ You could just sort the data by the target variable, and then assign every $n$th data to validation set, I think? $\endgroup$
    – Eoin
    Jan 12, 2021 at 18:20
  • $\begingroup$ What do you mean by "stratified shuffling"? The only relevant hit when searching it is the page from the sklearn documentation. I know "stratified sampling", but in that context, I don't understand the question. Is this in context of a classification you have already performed, or is this a theoretical concern? $\endgroup$
    – cherub
    Jan 13, 2021 at 21:06

1 Answer 1

Reset to default
2
+50
$\begingroup$

Stratified shuffling used in the case target variable takes discrete values. "The folds are made by preserving the percentage of samples for each class," stated in sklearn documentation. So your question is how I split my data with a continuous target variable such that distrb. is preserved.

One way to do it is bin your target variable:

  • Use sklearn KBinsDiscretizer to bin your target variable
  • Assign discrete labels to each bin
  • Use stratified shuffling, with the binned variable as the target to split your data

With this technique, you should be able to preserve mean-variance in each fold given the number of bins determined accordingly.

To determine the number of bins: https://stats.stackexchange.com/a/862/293623

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.