1
$\begingroup$

I'm trying to split my data into a testing and a training set. There are lots of variables that I want to ensure are well represented in both the training and testing sets (say, 15 covariates). But when I go to sample using rsample::initial_split(), it tells me: "Stratifying groups that make up [less than] 10% of the data may be statistically risky."

Theoretically, can someone explain why such a thing would be risky? I am aware that I can override this with pool = 0, I'm just trying to understand the concept here.

Update: Assume it is appropriate to do a test/train split based on the size of the data (many observations).

$\endgroup$
1
  • $\begingroup$ Unless you have tens of thousands of cases, just performing a train/test split is risky. You will lose precision in building the model and power for evaluating the model. See this post for a brief introduction. Internal validation of the modeling approach, for example by bootstrapping, is preferable with smaller data sets. See Harrell's Regression Modeling Strategies. $\endgroup$
    – EdM
    Commented Jan 23, 2023 at 18:26

1 Answer 1

0
$\begingroup$

I don't know exactly what the rsample authors had in mind. But I have seen other people ask them similar questions on their GitHub issues page, and it seems like it might be a computational concern more than a statistical one---especially when making strata in the first place. For example here:

In most cases (and examples that people have sent me), there are a lot of sparse or empty cells in the strata so it would be hard to automate this in a general way.

For an extreme example, imagine that your $n$ rows of data are organized into $n/2$ strata of $2$ observations each. Then you can't do data-splitting with any ratio other than 50/50 train/test splits; or you can't do $k$-fold CV with anything other than $k=2$.

Even if this split ratio is OK with you, perhaps the rsample authors are worried that this could cause other software problems. It might be hard to write the package in a way that lets the downstream rsample functions know to only allow 50/50 splits. If those downstream functions assume that you can always choose any reasonable split ratio or number of folds, they might break in unexpected and hard-to-debug ways if, say, you ask it for a 80/20 split when only 50/50 splits are actually feasible.


But here's a hypothetical example where it could also be statistically risky. Let's say you stratify on a continuous variable, and (as above) you break it into $n/2$ strata of 2 observations each. Then you randomly assign one case from each stratum to training and the other to testing.

If this variable is informative, this might cause your training and testing sets to be almost identical, despite the random partitioning within each stratum. In that case, your test set just wouldn't be very useful as holdout data. It would almost be as bad as if you had simply made 2 identical copies of your dataset, and used one to train and the other to test.

This kind of extreme example seems unlikely in practice if you are thinking carefully about your data. But I can also see why rsample warns you to stop and think before you proceed.

(That said, I'd be curious to hear other people's takes on why small strata might be a bad idea, and whether there's any reason to use 10% as the threshold.)

[Update: Of course if you really did originally sample the data using many small strata with sample sizes of 2 each, then you should indeed use those strata for sample splitting. Balanced Repeated Replication is a closely related idea, though it's used for standard error estimates rather than for model assessment & comparison.]

$\endgroup$
4
  • $\begingroup$ Thanks for your answer! I guess I don't follow why those pairs of observations would be identical...isn't the whole point of stratified sampling that you are ensuring like observations show up in both the testing and the training set? I guess what I'm saying is that observations in a given strata will be similar by design, but rarely identical. $\endgroup$
    – Aegis
    Commented Jan 23, 2023 at 19:33
  • $\begingroup$ Aaaah, but are you saying that the smaller the strata, the more similar the observations will be? That makes sense to me...but the response variable for those two observations could be really different--and we would never want to stratify by the response variable. $\endgroup$
    – Aegis
    Commented Jan 23, 2023 at 19:40
  • $\begingroup$ I don't mean that such pairs of observations would generally be identical. But there are cases where they might be. Say you just have two continuous variables, and there is some smooth true underlying relationship: Y = f(X) + noise. If your 1st stratum has the two smallest X values, the 2nd stratum has the next two smallest X values, and so on... then the Xs in each pair will be near-identical, and the Y-values in each pair might be quite close to each other too. (Again, it won't necessarily happen! It's just one possible situation where small strata might be statistically risky.) $\endgroup$
    – civilstat
    Commented Jan 23, 2023 at 19:55
  • $\begingroup$ Makes sense--thanks $\endgroup$
    – Aegis
    Commented Jan 23, 2023 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.