3
$\begingroup$

I have a huge dataset ($n$ around five million, $p$ around three thousand) for a classification problem, where my interest is predictive class probabilities for test data, not the target. I shall be using bootstrap samples to estimate the smoothed probabilities from various models.

The problem is when I bootstrap the cases not all the levels in the categorical predictors will be in every resample. Given that I need parameters for all levels when working with the test set, I'm unsure of what to do.

I read Agresti's Categorical Data Analysis, but there doesn't seem to be mention of this.

I have thought of 2 possibilities:

  1. Insert a base composition of varied cases in every resample so that all levels are included for all predictors.

  2. Define all levels for each categorical variable with reference to the data schema and run the models.

I'm having to think about this because my resamples are pulled as CSV data from a SQL database and I would usually use read.csv() which automagically handles levels, preference relations and levels for categorical data using the cases in the CSV file. (Pulling all of the data in one csv is not an option due to resource constraints.)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Stratified bootstrap sampling is perfectly permissible to use. If you have the proportions of the class levels in the population I think you just make the class level a stratum and sample cases with each strata proportional to their representation in the population. Stratified bootstrap sampling amounts to sampling with replacement a specified number of times in each stratum.

Edit to the original answer: One aspect of the problem that was not addressed originally when the question was asked was a sensible way of reducing the number of predictors. The value of stratification is to reduce the overall variation. With so many predictors some may not be very useful anyway and hence the number of levels could possibly be reduced without hurting the value of the predictors for classification seriously. To emphasize what I said in the original answer since the bootstrap samples with replacement from each individual stratum no strata will be left out in any bootstrap sample. I think this addresses the recent comment by @kjetilbHalvorsen.

Improve this answer
$\endgroup$
2
  • $\begingroup$ My data is such that to ensure at least one example of the lowest occurring level in every variable, a single bootstrap replicate size will exhaust my computer memory resources (64GB RAM and twice that swap file). This is because I have discrete-choice data, many predictors, mostly categorical and mostly with some categorical levels that have relatively low frequency. Would it introduce bias to cheat and ignore upcoming variables that have all levels included as the sample is built by stratifying on the earlier variables? I could then randomly decrease replicate size to something manageable. $\endgroup$
    – Yoda
    Commented Aug 18, 2012 at 17:13
  • $\begingroup$ @Michael Chernick: How will that work if there are many levels with very few observation, say, the extreme case with many levels with only one observation? is there not a risk that the variation induced by bootstrapping (with stratification) will not give sufficient variability? $\endgroup$
    – kjetil b halvorsen ♦
    Commented May 16, 2017 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.