Say I have two different datasets with a few hundred cases and 100 predictors each. The dependent variable in both dataset is the class (nominal) to which a case belongs, but in the first dataset, there are 15 such classes, whereas the second dataset only has 10 different classes. In principle, however, the cases in both datasets should be classified equally well into the correct class using the 100 available predictors. I grow a random forest (
mtry=10, ntree=1000) separately on both datasets and find that the out-of-bag (OOB) classification accuracy in the first dataset is, say, 75%, whereas it is 85% in the second dataset.
Question: Do I need inferential statistics to compare these two percentages, and if so, which? Is there any (subject-matter) study addressing this issue?
My thinking at the moment: I'm not sure what the reference populations that I want to generalise to are. On the one hand, I'm comparing the classification rates of two random forests, which are 'sampled' (due to the randomness inherent to the random forest algorithm) from a larger population of random forests. However, a sample of 10 random forests with
ntree=1000 is equivalent to a large random forest with
ntree=10000, and the classification rates are already at asymptote for
ntree=1000. Thus, if I want to generalise to the larger population of possible random forests grown on the same datasets, I think I don't need inferential statistics since I'm essentially dealing with population values.
On the other hand, the few hundred cases in each dataset are a sample of the population of possible cases, too. From this perspective, I should maybe compute a $\chi^2$ test on the number of correctly vs incorrectly classified cases in each dataset.