1
$\begingroup$

I am working on a stacked ensemble method. I trained three classifiers as my first-layer models and one Logistic Regression as my second layer model. I then stacked both the first-layer models and second-layer model and trained it as a Stacked Classifier. Now, I get a perfect score of 1.0 when I measure performance on my test set. This makes me suspicious.

Can you tell me, are these kind of models prone to over-fitting?

My code looks something like this:

# Creating the first-layer models
clf_knn =KNeighborsClassifier(n_neighbors= 5, algorithm= 'ball_tree')
clf_decision_t = DecisionTreeClassifier(min_samples_leaf = 5, min_samples_split = 
15, random_state=500)
clf_naive_b = GaussianNB()

# Creating second-layer model (meta-model)
clf_logistic_r = LogisticRegression()

# Creating and fitting the stacked model
clf_stack = StackingClassifier(classifiers=[clf_knn, clf_decision_t , clf_naive_b ], 
meta_classifier=clf_logistic_r )
clf_stack.fit(X_train, y_train)

# Evaluate the stacked model’s performance
print("Accuracy: {:0.4f}".format(accuracy_score(y_test, 
clf_stack.predict(X_test))))

I used scikit and mlxtend libraries

$\endgroup$
  • 1
    $\begingroup$ Could you please tell us more about the data, the problem and your validation methodology? Have you rerun those tests? Are you able to perform repeated k-fold for at least 50 cycles of training and testing? $\endgroup$ – Piotr Rarus - Reinstate Monica Dec 13 '19 at 11:27
  • $\begingroup$ Thank you for your answer. Indeed I should use cross-validation. Yohann suggested to use StackingCVClassifier , which is a newer version of StackingClassifier. It has cross-validation integrated. $\endgroup$ – Horbaje Dec 13 '19 at 13:19
1
$\begingroup$

Yes, it is not overfitting per se but because of information leakage. The mlxtend on StackingClassifier documentation also recommends using StackingClassifierCV instead which uses out-of-fold features to generate meta features. This essentially remove the leakage problem.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you ! I was not aware there exists a newer version of StackingClassifier that uses Cross-Validation. That should counteract over-fitting as described on the documentation page: rasbt.github.io/mlxtend/user_guide/classifier/… $\endgroup$ – Horbaje Dec 13 '19 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.