I don't know exactly what the rsample
authors had in mind. But I have seen other people ask them similar questions on their GitHub issues page, and it seems like it might be a computational concern more than a statistical one---especially when making strata in the first place. For example here:
In most cases (and examples that people have sent me), there are a lot of sparse or empty cells in the strata so it would be hard to automate this in a general way.
For an extreme example, imagine that your $n$ rows of data are organized into $n/2$ strata of $2$ observations each. Then you can't do data-splitting with any ratio other than 50/50 train/test splits; or you can't do $k$-fold CV with anything other than $k=2$.
Even if this split ratio is OK with you, perhaps the rsample
authors are worried that this could cause other software problems. It might be hard to write the package in a way that lets the downstream rsample
functions know to only allow 50/50 splits. If those downstream functions assume that you can always choose any reasonable split ratio or number of folds, they might break in unexpected and hard-to-debug ways if, say, you ask it for a 80/20 split when only 50/50 splits are actually feasible.
But here's a hypothetical example where it could also be statistically risky. Let's say you stratify on a continuous variable, and (as above) you break it into $n/2$ strata of 2 observations each. Then you randomly assign one case from each stratum to training and the other to testing.
If this variable is informative, this might cause your training and testing sets to be almost identical, despite the random partitioning within each stratum. In that case, your test set just wouldn't be very useful as holdout data. It would almost be as bad as if you had simply made 2 identical copies of your dataset, and used one to train and the other to test.
This kind of extreme example seems unlikely in practice if you are thinking carefully about your data. But I can also see why rsample
warns you to stop and think before you proceed.
(That said, I'd be curious to hear other people's takes on why small strata might be a bad idea, and whether there's any reason to use 10% as the threshold.)
[Update: Of course if you really did originally sample the data using many small strata with sample sizes of 2 each, then you should indeed use those strata for sample splitting. Balanced Repeated Replication is a closely related idea, though it's used for standard error estimates rather than for model assessment & comparison.]