5
$\begingroup$

So I was asked by a reviewer to provide the "P-value" for my random forest regression model.

I tried to do some research on this, and only found methods to produce p values for each split condition (like in 'party' package), and p-values for variable importance (like in 'rfPermute' package). I find it hard to trust the p-values for variable importance since some very important variables have p-values of >0.9.

Any input on how to generate the general "p-value" for random forest (if there is one) would be appreciated.

$\endgroup$
9
  • 2
    $\begingroup$ p-value of what? $\endgroup$
    – Firebug
    Commented Jun 6, 2017 at 19:16
  • $\begingroup$ So I reported the "% Var explained" and the reviewer asked me to provide the p-value. $\endgroup$
    – Xiaoyu Lu
    Commented Jun 6, 2017 at 19:25
  • 1
    $\begingroup$ @markwhite, I agree completely. It has always been my experience that you should not argue with reviewers, let alone try to educate them about stats as they will a) ignore you, b) reject the paper. Therefore it is much more easy to simply bootstrap on the raw data, and completely ignore what hypothetical distribution we are inferring about or whether the psudo R-square of an RF can be thought to come from anysuch distribution. But I digress. $\endgroup$
    – Repmat
    Commented Jun 6, 2017 at 19:50
  • 2
    $\begingroup$ The digression is appreciated; in my opinion, the peer-review process is one of the biggest hurdles in advancing statistical methodologies; many applied researchers don't continue their statistical education after finishing graduate school. $\endgroup$
    – Mark White
    Commented Jun 6, 2017 at 19:52
  • 1
    $\begingroup$ @MarkWhite: (obligatory snarky comment) some applied researchers appear to stop their statistical education after their undergraduate years and spend the rest of their academic life actively forgetting what they learned. All this while loudly proclaiming that they don't need no stinkin' statisticians for their applications or studies. $\endgroup$ Commented Jun 6, 2017 at 20:05

1 Answer 1

13
$\begingroup$

When in doubt, simulate or permute.

In this specific case:

  1. Randomly permute your dependent variable.
  2. Fit a random forest.
  3. Note the % variance explained.

Do steps 1-3 multiple times, say 1,000-10,000 times. You now have an empirical distribution of % variance explained through a random forest, under the null hypothesis of no relationship between your independent and dependent variable.

Insert the actual % variance explained in your original model into this distribution, and note which proportion of permutation-based "null" % variance explained values exceeds this true value. This proportion is your p value.

If you did the same thing in a standard linear regression model, you would (asymptotically) get the p value for the classical F test for variance explained.

As others write, your reviewer does not sound overly statistically savvy, but the approach I'm outlining above makes sense and should satisfy him. It's better than getting into an anonymous argument over the statistical competence of a reviewer, anyway.

$\endgroup$
11
  • $\begingroup$ What would be the steps for the simulation approach? $\endgroup$
    – Digio
    Commented Jul 11, 2019 at 8:16
  • $\begingroup$ @Digio: I'd go with the permutation approach in this case. $\endgroup$ Commented Jul 11, 2019 at 8:17
  • $\begingroup$ Stephan - I still have 1 question on this method. How would you estimate the % variance explained per predictor from the RF importance score? $\endgroup$
    – Digio
    Commented Jul 26, 2019 at 19:49
  • $\begingroup$ i would not calculate % variance explained from the RF importance score. I would assess % variance explained from the permutation analysis (by noting residual variance from the full model, and variance explained in a model with one predictor permuted), then define this as the variable's importance. Alternatively, use any other reasonable KPI, like Gini impurity reduction. $\endgroup$ Commented Jul 27, 2019 at 17:34
  • $\begingroup$ I see, thanks. By the way, by "RF importance score" I did mean Gini impurity or information gain. $\endgroup$
    – Digio
    Commented Jul 28, 2019 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.