4
$\begingroup$

For hypothesis test-bases statistical inference, one must ensure that the sample size is large enough to achieve sufficient power to reject the null hypothesis assuming that it is false. This is a critical consideration when designing studies, as there is no point in performing a study if there is only a small chance that an effect of interest can be detected.

The typical use case for random forest (RF) models does not involve hypothesis testing. Typically one's objective is either to maximize classification accuracy or minimize prediction error. For this reason, a power analysis as typically conceived is not meaningful in the case of an RF based analysis, as there is no null to reject. However, the principle of power still applies, namely, what guarantee do we have that our sample size is large enough so we can detect some signal of interest, assuming it's there?

I'll note that this is a very broad question, and that as I've phrased it there isn't a straightforward or even well defined answer. What I'm hoping to find here is a body of literature addressing this topic. I'm looking for guidance about how to think about the problem, or some papers where a power-analysis-like procedure has been performed for an RF model, or really any statistical procedure where sample size is relevant and that doesn't involve hypothesis testing. The answer may be as simple as a googleable term that I just don't know so I can research the question.

Thank you in advance!

$\endgroup$

3 Answers 3

1
$\begingroup$

The following paper seems like it might help get at your question. You could do a search for papers that cite this paper; if there’s a method for power analyses, they probably cited this one.

Mentch, L, Hooker, G (2016) Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research 17(1): 841–881.

$\endgroup$
1
  • $\begingroup$ JMLR is solid reference. $\endgroup$ Commented Sep 30, 2022 at 19:31
1
$\begingroup$

I think this is a question a lot of people writing NIH grants face as they require formal power analysis. I agree that it's hard to find formalized answers for this, but perhaps because it hasn't been formally addressed (think of all the different ways a RF model can be constructed!) At it's heart, power is that you're correctly rejecting the null hypothesis. That's the RF's true positive rate. You can estimate this with cross validation as a function of training set size. This is effectively a sample size estimation via learning curve. People have researched this to some degree, but it is still an emerging field in my opinion. Until there is something formalized for every model architecture, I think this would be a satisfactory general approach. This paper goes a bit further with this idea:

Figueroa, R.L., Zeng-Treitler, Q., Kandula, S. et al. Predicting sample size required for classification performance. BMC Med Inform Decis Mak 12, 8 (2012). https://doi.org/10.1186/1472-6947-12-8

$\endgroup$
-1
$\begingroup$

I'm lazy, and by lazy I mean greedy, and by lazy or greedy I mean that I have so much to do and so little time that "good enough" is "best".

For a problem like this a "cheap-shot" is to say "would a classic linear model work here", and if there is enough power that a linear model could do the job then I can infer that a more capable learner also has sufficient samples.

I also like to think about binomial confidence intervals, and if I have enough samples such that the CI size for a binomial "weighted coin" is smaller than my acceptable uncertainty, I like to use that.

I'm a bit of a glutton, and when I can get more data, I generally try to start with more and then start trimming.

$\endgroup$
2
  • 1
    $\begingroup$ (-1) I wish this were true but I don't see any basis for this claim: "if there is enough power that a linear model could do the job then I can infer that a more capable learner also has sufficient samples". The word 'capable' is doing a lot of work here, as it implies that a random forest is necessarily more powerful than a linear model. Which is an idea that deserves to be developed, at the very least. Under what conditions? $\endgroup$
    – mkt
    Commented Sep 30, 2022 at 19:44
  • $\begingroup$ @mkt - I was thinking of a regression tree model with linear interpolation, not constant interpolation, so a stump is really a linear regression. Splits are made to reduce the variance, so a 2-leaf model would be piecewise linear with the "elbow" placed to reduce the variance. When measuring least squared error between true and estimated, this kind of model can always equal the linear model, and in many cases out-perform it. $\endgroup$ Commented Sep 30, 2022 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.