I want to estimate the accuracy of a machine learning model. I have an independent test set with a vector of trusted labels and a corresponding vector of model-based predictions. If I assume the test set examples are independent, then I can bootstrap the test set and estimate a confidence interval around my accuracy score. However, there is a hierarchical structure to my data: the test set includes multiple examples from a number of clusters (people). So I need to adapt my estimation approach.
I have read about the cluster bootstrap and it seems that I should be bootstrapping (sampling with replacement from) the clusters rather than the examples. However, I have some practical questions about this.
- Should I be calculating one accuracy score across all examples in a resample or calculating accuracy per cluster and then aggregating these estimates somehow?
- Is there a way that my ultimate confidence interval can incorporate information about the variability at both levels of the hierarchy, both within and across clusters?
- Should I be sampling the same number of examples (without replacement) from each cluster to ensure that all clusters are equally represented in each resample or is it ok that resamples vary in size?
Much of the resampling literature focuses on linear models so I'm not exactly sure how to adapt this to something like accuracy (which is simpler in many ways).
