I've looked at some other questions on bootstrap significance testing:
- Non-parametric bootstrap p-values vs confidence intervals
- Computing p-value using bootstrap with R
- p-value vs. confidence interval obtained in Bootstrapping
...and mine is a bit different. I think my problem might be specific to cases where the parameters under the null are on the boundary of the parameter space.
Suppose you do a case-resampling bootstrap, where you resample observations with replacement, fit a model, and get a bootstrap distribution for the parameter you are trying to estimate.
Now suppose the parameter and your estimator are strictly positive. For example, REML estimates of variance of random effects in random/mixed effect models, or estimates of information gain in classification trees. I have found often that often the canonical null hypotheses in these cases is is that the parameter = 0.
Example: Random Effects Modeling
I'll use a random effects model as a specific example. A parametric bootstrap to test the hypothesis that the variance of the random effect is 0 might go as follows:
- Fit the model with the random effect using maximum likelihood.
- Do the same for the model without the random effect.
- Calculate a likelihood test statistic
- Repeat steps 1:3 using a response simulated with resampled errors. Do this n times, obtaining a bootstrap distribution of likelihood test statistics
- Calculate the bootstrap p-value of the original test-statistic using the bootstrap distribution.
Now suppose you want to use a REML estimator. The model with and without the random effect are not comparable using the likelihood test. So why not try a case-resampling bootstrap? In a case-resampling bootstrap, one would obtain a bootstrap distribution of variance estimates for that random effect using data resampled with replacement. But then what?
If we were trying to do inference on a regression coefficient, one might conclude in favor of the null that a regression coefficient is 0 if the bootstrap confidence interval contains 0. But this is never the case for an estimator that is >= 0. How does one evaluate the significance of the estimate in this case?