Evaluating Feature Importance for a Super Learner Ensemble Meta-Model

Question

I have been reading up on super learner ensemble methods that utilize multiple models and model configurations to make model predictions as good or better than any individual base model previously investigated. Machine Learning Mastery has a great tutorial on this topic.

I have tested and evaluated a super learner ensemble algorithm for a specific classification task and the performance of this meta-model is superior to any individual base model I had previously evaluated. Model interpretability is important for my use-case and I have two primary questions/concerns prior to deploying the super learner ensemble meta-model I have developed:

1. The super learner meta-model I have developed is comprised of multiple sub-models trained on different feature sets (i.e. differences in preprocessing/use of encoded data vs. categorical data for multiple sub-models). How does one appropriately evaluate feature importance for the overarching meta-model when individual sub-models are likely to have differences not only in the feature sets each model was trained on but also which features are most important in making predictions? I have previously used SHAP for evaluation of feature importance using Shapley additive explanations for individual models but am not sure how to apply a similar evaluation for a meta-model built on top of multiple different models?

2. Unless there is a better alternative, deploying a super learner ensemble algorithm comprised of multiple sub-models trained on different feature sets would require preprocessing incoming data separately for each individual sub-model before the overarching meta-model is able to make predictions based on the collective predictions from the sub-models. Is this the standard approach?

Answer 1

1. shap's kernel explainer works on an arbitrary model, taking the prediction function and a background/reference dataset as inputs; using the ensemble's prediction function should work fine.

2. I've often seen a single preprocessing done to all the data, passing that to each base model. In the k-fold prediction happening here, that can cause some data leakage, but how much depends on what kind of preprocessing you're doing. And, the leakage only impacts the overfitting of the base models' predictions given to train the meta-model, not the final base models nor the test score on some held-out data (not seen even by the preprocessor).

   If you want to use more powerful preprocessing, or completely avoid leakage, then yes you should attach preprocessing in a pipeline to each base model. You might be able to use caching to reduce some redundant calculations.

Finally, note that sklearn has its own stacking estimators now, and they use the k-fold base prediction method by default.