# StackingClassifier: Different Feature Sets

I have a few different feature sets (so, with same number of rows and the labels are the same) that I use for different ML models. In my case these sets are DataFrames.

I want to use them to train a StackingClassifier. But the fitting step only allows to specify a single feature set. Goal is to fit clf1 with df1 and clf2 with df2, etc.

stack_clf = StackingClassifier(estimators=[('df1-clf', clf1), ('df2-clf', clf2)],final_estimator=LogisticRegression())
stack_clf.fit(...)


How should I proceed with this kind of situation? I have been trying to find a solution using pipelines but am not sure if that is the way to go. There was a similar question several years ago with voting classifiers, but at the time the solution was apparently not natively supported by SKlearn, and the custom functions suggested were specific to voting classfiers. Thanks in advance for any help.

StackingClassifier does not support training each base estimator on different feature sets. I do not see how you could use a Pipeline to cope with this either. With its current implementation, you need to call the fit method with one feature set for all classifiers because StackingClassifier will otherwise not expose attributes ending a trailing _ which are needed to identify the classifier as fitted.

If you are not bound to use scikit-learn, an easy way out would be to use the h2o library and train a H2OStackedEnsembleEstimator. As written in the documentation, it explicitly supports training each base estimator on different feature sets:

The models must be trained on the same training_frame. The rows must be identical, but you can use different sets of predictor columns, x, across models if you choose.

An example for Python can be found here.

• Thanks - really appreciate your answer. I will look into the H2O library.
– Bow
Aug 22, 2021 at 4:07

There's actually a way to pull this off using Pipeline base estimators with simple custom transformers.

1. Create a custom transformer for each classifier. This will be a simple transformer simply slicing the data. It can be achieved by using a lambda function inside sklearn's FunctionTransformer.
2. Create a two-step Pipeline for each base estimator with its corresponding transformer
3. Create a StackingEstimator with its base estimators being the Pipelines from (2)
    from sklearn.datasets import make_classification
from sklearn.ensemble import StackingClassifier
from sklearn.preprocessing import FunctionTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline

# Dummy data, X is the super set containing all features:
X, y = make_classification(n_features=10)

# Base estimator 1 uses the first 3 features with a logistic regression:
clf_1 = LogisticRegression()
clf_1_transformer = FunctionTransformer(lambda X: X[:, :3])
tclf_1 = Pipeline(
[('transformer_1', clf_1_transformer), ('clf_1', clf_1)]
)

# Base estimator 2 uses the last 7 features with a SVM:
clf_2 = LinearSVC()
clf_2_transformer = FunctionTransformer(lambda X: X[:, 3:])
tclf_2 = Pipeline(
[('transformer_2', clf_2_transformer), ('clf_2', clf_2)]
)

# The meta-learner uses the transformed-classifiers as base estimators:
sclf = StackingClassifier([('tclf_1', tclf_1), ('tclf_2', tclf_2)])
sclf.fit(X, y)

# Sanity check: check how many coefficients each estimator uses:
print(sclf.estimators_[0].steps[-1][1].coef_.size)  # 3
print(sclf.estimators_[1].steps[-1][1].coef_.size)  # 7


Note that a more proper way to create the column-selection transformer might be to inherent from BaseEstimator and TransformerMixin instead of hacking the FunctionalTransformer.