I don't understand how features stacking works. I found out the following sample guideline:

  1. Split the train set in 2 parts: train_a and train_b

  2. Fit a first-stage model on train_a and create predictions for train_b

  3. Fit the same model on train_b and create predictions for train_a

  4. Finally fit the model on the entire train set and create predictions for the test set.

  5. Now train a second-stage stacker model on the probabilities from the first-stage model(s).

Now let's take an example of a multi-class classification with 4 classes. In step 2 I get 4 columns (each one corresponding to the class probability), e.g.

Class1 Class2 Class3 Class4
0.22   0.58   0.05   0.15

Then in Step 3 I get another 4 columns. I assume that after this I should take the probability predictions obtained in Step 3 and append them to 2 in order to get a matrix with 4 columns.

How I should use this intermediate result in Steps 4 and 5?


1 Answer 1


Quick Answer

Use Mlxtend's StackingCVClassifier to perform feature stacking with any number of Sklearn classifiers.

See: http://rasbt.github.io/mlxtend/user_guide/classifier/StackingCVClassifier/

A Deeper Explanation

In feature stacking you typically have 2 or more level 1 classifiers and one "meta" classifier.

It is my understanding that the level 1 classifiers are used to create a new training dataset, typically with predicted probabilities. In Sklearn for example, many classifiers will have a predict_proba() function.

The "meta" classifier takes the predicted probabilities from the 2 or more level 1 classifiers as training data to build the final model.

Cross validated feature stacking extends the algorithm ensuring that the predicted probabilities or predictions used to build the meta classifier's training dataset come from records that the level 1 classifiers have not seen before. This avoids over fitting due to information leakage between the level 1 and meta data classifier.

A Very Basic Example

So how might a VERY simple example work in Sklearn? I wrote this in about 10 minutes. I am sure someone else could do a better job (like mlxtend)!

The following example assumes you use 10-fold cross validation and 3 level 1 classifiers. Each iteration of the loop uses 90% of the data to train the 3 level 1 classifiers on Sklearn's 20 newsgroups dataset. The remaining 10% of the test data is used to create predicted probabilities for each of the 3 level 1 classifiers.

So the dataset passed to the meta-classifier has 21 features. This is because the 20 newsgroups dataset has 7 target classes * 3 level 1 classifiers producing 7 predicted probabilities for each prediction made. I use numpy's hstack function to combine the 3 level 1 classifier's predicted probabilities and the vstack function to union all of the 10-folds of test records back into 1 training dataset for the meta classifier.

Finally, I use Sklearn's cross_validate() function along with the meta classifier (LogisticRegression in this example) to perform one final 10 fold cross validation on the dataset created by the level 1 classifiers.

Get the 20 Newsgroups Data

from sklearn.datasets import fetch_20newsgroups

categories = [

data, target = fetch_20newsgroups(subset='train', categories=categories,

X = data
y = target

Vectorize the Data into Something we can Build a Model with using TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(analyzer='word', ngram_range=(1,3), binary=False, use_idf=False, norm=None)
X_tfidf = tfidf.fit_transform(X)

Building the new Level 1 Training Dataset

Here we use three sklearn classifiers: MultinomialNB, SGDClassifier, and RandomForestClassifier to create the level 1 training dataset. We train on 90% of the data, then get a total of 21 predicted probabilities (1 for each of the 7 20 newgroup target classes) from the 3 classifiers using 10% of the data that StratifiedKFold has set aside for each test fold. I merge these predicted probabilities together using numpy's hstack (for the columns from each classifier's predicted probs) and vstack (for the test records in each of the 10 folds). After this runs l2_x and l2_y contain the new training dataset that we will use for our meta classifier.

import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import SGDClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_validate

k = 10

l1_classifiers = [
    SGDClassifier(loss='log', penalty='l2',
                       alpha=1e-3, random_state=42,
                       max_iter=5, tol=None),
    RandomForestClassifier(n_estimators = 500, criterion = "gini", max_depth = 10,
                             max_features = "auto", min_samples_leaf = 0.005,
                             min_samples_split = 0.005, n_jobs = -1, random_state = 42)

cv = StratifiedKFold(n_splits=k, shuffle=True, random_state=42)

l2_X = []
l2_y = []

for i in range(k):

    train, test = next(cv.split(X_tfidf, y))

    l2_X_probs = []

    for clf in l1_classifiers:
        # Train each of the level 1 classifiers
        # Create the level 2 dataset

    # Combine predicted probs for all classifiers
    X_l2_k = np.hstack(l2_X_probs)
    # Append our X probs and y to the new l2 dataset

# Vstack the new X and y data for all of our folds
l2_X = np.vstack(l2_X)
l2_y = np.vstack(l2_y)

Validation of the Meta Classifier

For the example, I chose LogisticRegression as the meta classifier. I perform stratified 10-fold cross validation one final time on the new dataset created by the l1 classifier's predicted probabilities.

# Create the meta classifier
meta_clf = LogisticRegression(random_state=42) 

# Validate the new training dataset and the meta classifier
cv_results = cross_validate(meta_clf, l2_X, l2_y, cv=cv, scoring="accuracy", n_jobs=-1)

print ('Fold Scores:')
print(' ')
print(' ')
print('Mean Accuracy: ', cv_results['test_score'].mean())
print('Mean Fit Time: ', cv_results['fit_time'].mean())
print('Mean Score Time: ', cv_results['score_time'].mean())

Using the Model in a Production Environment

If you wanted to use the final meta classifier in a production environment, you will still need each of your fitted l1 classifiers to call predict_proba() on and assemble the input data into the correct format that your meta classifier accepts! This is why it is easier to use a library that does all of this for you behind the scenes like mlextend's StackingCVClassifier.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.