I have been reading up on super learner ensemble methods that utilize multiple models and model configurations to make model predictions as good or better than any individual base model previously investigated. Machine Learning Mastery has a great tutorial on this topic.
I have tested and evaluated a super learner ensemble algorithm for a specific classification task and the performance of this meta-model is superior to any individual base model I had previously evaluated. Model interpretability is important for my use-case and I have two primary questions/concerns prior to deploying the super learner ensemble meta-model I have developed:
The super learner meta-model I have developed is comprised of multiple sub-models trained on different feature sets (i.e. differences in preprocessing/use of encoded data vs. categorical data for multiple sub-models). How does one appropriately evaluate feature importance for the overarching meta-model when individual sub-models are likely to have differences not only in the feature sets each model was trained on but also which features are most important in making predictions? I have previously used SHAP for evaluation of feature importance using Shapley additive explanations for individual models but am not sure how to apply a similar evaluation for a meta-model built on top of multiple different models?
Unless there is a better alternative, deploying a super learner ensemble algorithm comprised of multiple sub-models trained on different feature sets would require preprocessing incoming data separately for each individual sub-model before the overarching meta-model is able to make predictions based on the collective predictions from the sub-models. Is this the standard approach?