What is the current thinking on performing power analyses when using a random forest model? For hypothesis test-bases statistical inference, one must ensure that the sample size is large enough to achieve sufficient power to reject the null hypothesis assuming that it is false. This is a critical consideration when designing studies, as there is no point in performing a study if there is only a small chance that an effect of interest can be detected.
The typical use case for random forest (RF) models does not involve hypothesis testing. Typically one's objective is either to maximize classification accuracy or minimize prediction error. For this reason, a power analysis as typically conceived is not meaningful in the case of an RF based analysis, as there is no null to reject. However, the principle of power still applies, namely, what guarantee do we have that our sample size is large enough so we can detect some signal of interest, assuming it's there?
I'll note that this is a very broad question, and that as I've phrased it there isn't a straightforward or even well defined answer. What I'm hoping to find here is a body of literature addressing this topic. I'm looking for guidance about how to think about the problem, or some papers where a power-analysis-like procedure has been performed for an RF model, or really any statistical procedure where sample size is relevant and that doesn't involve hypothesis testing. The answer may be as simple as a googleable term that I just don't know so I can research the question.
Thank you in advance!
 A: The following paper seems like it might help get at your question. You could do a search for papers that cite this paper; if there’s a method for power analyses, they probably cited this one.
Mentch, L, Hooker, G (2016) Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research 17(1): 841–881.
A: I think this is a question a lot of people writing NIH grants face as they require formal power analysis. I agree that it's hard to find formalized answers for this, but perhaps because it hasn't been formally addressed (think of all the different ways a RF model can be constructed!) At it's heart, power is that you're correctly rejecting the null hypothesis. That's the RF's true positive rate. You can estimate this with cross validation as a function of training set size. This is effectively a sample size estimation via learning curve. People have researched this to some degree, but it is still an emerging field in my opinion. Until there is something formalized for every model architecture, I think this would be a satisfactory general approach. This paper goes a bit further with this idea:
Figueroa, R.L., Zeng-Treitler, Q., Kandula, S. et al. Predicting sample size required for classification performance. BMC Med Inform Decis Mak 12, 8 (2012). https://doi.org/10.1186/1472-6947-12-8
A: I'm lazy, and by lazy I mean greedy, and by lazy or greedy I mean that I have so much to do and so little time that "good enough" is "best".
For a problem like this a "cheap-shot" is to say "would a classic linear model work here", and if there is enough power that a linear model could do the job then I can infer that a more capable learner also has sufficient samples.
I also like to think about binomial confidence intervals, and if I have enough samples such that the CI size for a binomial "weighted coin" is smaller than my acceptable uncertainty, I like to use that.
I'm a bit of a glutton, and when I can get more data, I generally try to start with more and then start trimming.
