I would like to use Permutation Feature Importance (PFI) with a set of different models coming from a bootstrap procedure.

I was looking at the algorithm here and was wondering how it can be used when the model is not just one.


  • Compute the reference score $s$ of the model $m$ on data $D$ (for instance the accuracy for a classifier or the $R^2$ for a regressor).

  • For each feature $j$ (column of $D$):

    • For each repetition $k$ in $1,...,K$:

      • Randomly shuffle column $j$ of dataset $D$ to generate a corrupted version of the data named $\tilde{D}_{k,j}$.

      • Compute the score $s_{k,j}$ of model $m$ on corrupted data $\tilde{D}_{k,j}$.

    • Compute importance $i_j$ for feature $f_j$ defined as $i_j = s-\frac{1}{K}\sum{_{k=1}^{K}s_{k,j}}$

I can think of a simple way which is taking the mean score per feature from every model but I am not sure if this can be done or if there are other ways to do that.

  • 1
    $\begingroup$ Taking the mean PFI of all the models is probably the easiest option. $\endgroup$ Commented Sep 26, 2023 at 8:33
  • $\begingroup$ Thanks! I was wondering if there was any paper or book where they did this before, but it makes sense. $\endgroup$
    – umbe1987
    Commented Sep 26, 2023 at 9:59
  • 1
    $\begingroup$ As shown at stats.stackexchange.com/questions/626376 this permutation measure does not perform correctly when there is strong multicollinearity. $\endgroup$ Commented Sep 26, 2023 at 10:37
  • $\begingroup$ Thanks, I am aware of this and I am performing variable selection and avoid multicollinearity before this step. $\endgroup$
    – umbe1987
    Commented Sep 26, 2023 at 11:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.