Bootstrap validation with a categorical outcome: should I sample each outcome separately? I am doing something like what rms::validate does: bootstrap data frame rows in a supervised learning problem, fit a model to each bootstrap sample, apply that bootstrap model to the entire data set, and calculate performance.
I have a categorical outcome. When I do my bootstrap sampling, should I bootstrap group $0$ and group $1$ separately and then stitch them together when I got to fit my model, ensuring that the class ratio is constant for each bootstrap iteration, or should I bootstrap the entire data frame, allowing the class ratio to fluctuate? I think Harrell's rms::validate function does the latter.
I am interested in hearing arguments for and against each approach.
 A: I think the latter is the way to go, that is, bootstrapping (i.e., drawing $n$ observations from the original sample with replacement) without imposing any restriction on the class ratio.
My logic is that this resampling mechanism provides us with a Maximum-Likelihood Estimator (MLE) for the probability distribution $F$ from which our sample is drawn. Such estimator consists of the empirical probability distribution $\hat{F}$ that puts probability $1/n$ on each point. Thus, by imposing restrictions on the resulting class ratio, we implicitly put different probabilities on different observations, thus estimating $F$ in a different way.
I would like to specify that in my reasoning I assumed iid sampling and $n$ big enough, so that the original sample is representative of the population. To put thing more formally, I assume the existence of a probability distribution $F$ from which we draw $n$ iid observations. Then, I claim that the right way to proceed is to draw with replacement from $\hat{F}$ as defined above, without caring about the resulting class ratio.
For further details, I recommend Computer Age Statistical Inference (Efron and Hastie, 2016) and references therein. In particular, chapter 10 explains both parametric and nonparametric bootstrap, and pages 159-160 support my claim.
Anyway, I am really interested in other opinions, since I never actually thought of this.
