Best statistical test for comparing two random forest models, where each has a different set of variables available for modeling?

Question

What is the best statistical test for comparing two random forest models, where a different set of variables is made available for each model? I need a test where a power test can be used to justify a sample size. I'm considering using a paired t-test (see below).

More specific information follows.

Two random forest models are generated as follows. The data used is from a stratified random sampling from a 100x100 grid. The same sampled data is used to compute two different random forest models. One random forest model uses optimal variables (the variables used in data generating process). The other random forest model has more variables available (optimal, sub-optimal, and irrelevant). Multiple re-samplings could be done on the same grid; multiple random forest models could be generated using the same data; and more grids can be generated from the same stochastic data generating process. At least 20 different grids will be generated.

I'm considering using a paired t-test, and doing only only sampling for each grid, and computing only one pair of random forest models for each sampling. The power test would tell me, I believe, how many grids I need to generate. The R function pwr.t.test computes the power of a paired t-test. You give pwr.t.test all but one (any one) of the following values, and it gives you the one you left out: sample size, effect size, alpha (p-value), and power. I'm considering using AUC-ROC as the metric.

Is there a better comparison methodology? I need to be able to use a power test to justify a sample size.

The data generating process. I am generating simulated virtual species distributions. Independent variables are measured at different scales. Scale: how many cells around a given cell are averaged to determine the value of an environmental variable, where the averaged value is used in determining whether a grid point is a presence or absence. Each environmental variable is one layer of the grid, and is generated using functions available in a virtual SDM modeling package. A species is present or absent based on the combined values of the environmental variables at a grid location, based on a defined suitability function.

Answer 1

You may find yourself interested in Benavoli et al. (JMLR 2017). In addition to making a case for the Bayesian methods the authors like, the paper goes through more classical methods from frequentist statistics that you may prefer.

I do not see the relevance of the model construction when it comes to testing. While the fact that you are using different data sources to construct the models might have enormous implications for your work, when it comes to testing, you just have two models that you want to compare, no different than if you wanted to compare a random forest to a neural network.

Benavoli et al. (JMLR 2017) measure their models using classification accuracy, but their techniques do not seem to depend on the measure of performance on which the models are compared. If you have reason to want to investigate the ROCAUC, do feel free to use that in the methods in the article.

If you need to do power calculations, simulation can be a powerful tool for these kinds of tests where analytical solutions are not straightforward or necessarily even possible.

REFERENCE

Benavoli, Alessio, et al. "Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis." The Journal of Machine Learning Research 18.1 (2017): 2653-2688.