See the following:

enter image description here

From : https://scikit-learn.org/stable/modules/permutation_importance.html

The part I'm unsure about is:

Its validation performance, measured via the score, is significantly larger than the chance level

R2 measures the proportion of variance explained by the model, but why should a particular value (in this case 0.356) be considered higher than chance?

  • 2
    $\begingroup$ It means that you do a better job of predicting than always guessing the mean of the training values, regardless of input. $\endgroup$ – Dave Oct 26 '20 at 23:58
  • 1
    $\begingroup$ Here, "chance" refers to the model where $Y$ and $X$ are truly unrelated. In the permutation model, there is no systematic relationship between $Y$ and $X$; ie they are unrelated in reality. Nevertheless, by chance alone, every permutation sample will show a random trend, and hence a nonzero Rsquared that is explained purely by chance. Over 1000s of such samples, the distribution of these chance-only Rsquares is found. If your observed value, .356, is larger than most of these (say 95%) values, then it can be said to be "higher than chance," although the phrase is awkward. $\endgroup$ – BigBendRegion Oct 28 '20 at 11:41
  • $\begingroup$ @BigBendRegion thanks, so here they've just asserted that, rather than shown it. They've also stated this prior to doing anything with permutation importance, so I guess my question is - if i was to try and demonstrate that this found R2 value was indeed greater than chance, how would I go about doing so? We've got a single test/train split here and a single model, how do I know (from that output) that the given R2 value is greater than chance? I hope that's clear, cheers $\endgroup$ – baxx Oct 31 '20 at 13:35
  • 1
    $\begingroup$ Start here: 1. You need the distribution of R2 values - one for each permutation. Best if there are 1000s of them. 2. Draw the histogram of these R2 values. 3. Locate .356 on the horizontal axis. 4. Is it larger than "most" of the histogram? If yes, then in can be characterized as "greater than chance", because of the R2 values in the histogram are explained by chance. (If .356 happened to be well within the range of the chance-only R2 values, then your R2 can be characterized correctly as explainable by chance alone. $\endgroup$ – BigBendRegion Oct 31 '20 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.