What is the best statistical test for comparing two random forest models, where a different set of variables is made available for each model? I need a test where a power test can be used to justify a sample size. I'm considering using a paired t-test (see below).
More specific information follows.
Two random forest models are generated as follows. The data used is from a stratified random sampling from a 100x100 grid. The same sampled data is used to compute two different random forest models. One random forest model uses optimal variables (the variables used in data generating process). The other random forest model has more variables available (optimal, sub-optimal, and irrelevant). Multiple re-samplings could be done on the same grid; multiple random forest models could be generated using the same data; and more grids can be generated from the same stochastic data generating process. At least 20 different grids will be generated.
I'm considering using a paired t-test, and doing only only sampling for each grid, and computing only one pair of random forest models for each sampling. The power test would tell me, I believe, how many grids I need to generate. The R function pwr.t.test computes the power of a paired t-test. You give pwr.t.test all but one (any one) of the following values, and it gives you the one you left out: sample size, effect size, alpha (p-value), and power. I'm considering using AUC-ROC as the metric.
Is there a better comparison methodology? I need to be able to use a power test to justify a sample size.
The data generating process. I am generating simulated virtual species distributions. Independent variables are measured at different scales. Scale: how many cells around a given cell are averaged to determine the value of an environmental variable, where the averaged value is used in determining whether a grid point is a presence or absence. Each environmental variable is one layer of the grid, and is generated using functions available in a virtual SDM modeling package. A species is present or absent based on the combined values of the environmental variables at a grid location, based on a defined suitability function.