# Permutation Feature Importance with an ensemble of models

I would like to use Permutation Feature Importance (PFI) with a set of different models coming from a bootstrap procedure.

I was looking at the algorithm here and was wondering how it can be used when the model is not just one.

$$Algorithm$$

• Compute the reference score $$s$$ of the model $$m$$ on data $$D$$ (for instance the accuracy for a classifier or the $$R^2$$ for a regressor).

• For each feature $$j$$ (column of $$D$$):

• For each repetition $$k$$ in $$1,...,K$$:

• Randomly shuffle column $$j$$ of dataset $$D$$ to generate a corrupted version of the data named $$\tilde{D}_{k,j}$$.

• Compute the score $$s_{k,j}$$ of model $$m$$ on corrupted data $$\tilde{D}_{k,j}$$.

• Compute importance $$i_j$$ for feature $$f_j$$ defined as $$i_j = s-\frac{1}{K}\sum{_{k=1}^{K}s_{k,j}}$$

I can think of a simple way which is taking the mean score per feature from every model but I am not sure if this can be done or if there are other ways to do that.

• Taking the mean PFI of all the models is probably the easiest option. Commented Sep 26, 2023 at 8:33
• Thanks! I was wondering if there was any paper or book where they did this before, but it makes sense. Commented Sep 26, 2023 at 9:59
• As shown at stats.stackexchange.com/questions/626376 this permutation measure does not perform correctly when there is strong multicollinearity. Commented Sep 26, 2023 at 10:37
• Thanks, I am aware of this and I am performing variable selection and avoid multicollinearity before this step. Commented Sep 26, 2023 at 11:14