Fitting Stacking Classifer when Underlying Models Use Different Feature Subsets I’m looking to create a stacking classifier based on 3 underlying algorithms. I have already performed feature selection on each of the 3, and each returned a slightly different subset of features for optimal performance.  My issue is I’m now unsure which set of these various x_train values to use to fit the stacking classifier – one of the three subsets, or the full set of features I started with.
I’m new to ensemble models so apologies if this is a basic question, but couldn’t find any previous posts on the topic. Any responses appreciated.
 A: The ensemble works by having multiple models and combining their predictions in some way (e.g. a weighted average or some more complex model for combining them). Each of the different initial models that get combined can have different inputs. I.e. they don't need to use the same input features, at all.
If you believe you have created features that work particularly well for one model (e.g. for xgboost you target encoded categorical variables, while for a neural network you used an embedding layer and for random forest you one-hot-encoded; or for linear regression you did kNN imputation and for LightGBM you rely on the build-in method for missing data) or selected a subset of features that works well for each model, then you should probably just give each model those inputs. In fact, it can to some extent be helpful when the different models differ in a lot of ways (as long as they perform still decently well, even better if each performs best with very different approaches in several respects), because ensembling is usually the most effective the more different the predictions of the models are (of course still with the limitation that they need to be somewhat close in performance - adding very bad models that differ a lot does not usually help).
In practice, you might either have a super-set of inputs that then can be fed into all models (that in a way feels neater), or you have a separate training (and validation & test) data derivation for each model (quite common e.g. when in a team in a machine learning competition several team members write their own code and then ensembling is done on top of that - unless runtime becomes an issue it is usually not worth it to refactor the different codebases into a unified one).
