In my model testing, I tried to use model ensembling (blending in this case) to get better results. However the ensemble cannot beat single RandomForrestClassifier.

In first layer, I train meta-estimators to create train and test prediction sets

Then in second layer I train another set of models on those prediction sets and combine the predictions in single one.

I think I have there two things which worsen the prediction.

  • When combining predictions in 1st layer for blend_test I am using mean

  • Same with 2nd layer, when combining predictions for b_pred_m, I am using mean too. I tried weighted, but it did not help.

I would appreciate any tips how to improve my ensemble model.

def run():
   n_folds = 
   clfs = [RandomForestClassifier(n_estimators = 200, class_weight={0:1,1:2}),
            ExtraTreesClassifier(n_estimators = 200, criterion = 'gini'),
            GradientBoostingClassifier(n_estimators = 200),
    clfs_name = ['RandomForestClassifier', 'ExtraTreesClassifier', 
                 'GradientBoostingClassifier', 'KNeighborsClassifier', 
                 'LogisticRegression', 'GaussianNB']

    # Ready for cross validation
    skf = list(StratifiedKFold(y_train, n_folds))

    # Pre-allocate the data
    blend_train = np.zeros((X_train.shape[0], len(clfs))) # Number of training data x Number of classifiers
    blend_test = np.zeros((X_test.shape[0], len(clfs))) # Number of testing data x Number of classifiers
    for j, clf in enumerate(clfs):
        print 'Training classifier: %s' % clfs_name[j]
        # Number of testing data x Number of folds , 
        # we will take the mean of the predictions later
        blend_test_j = np.zeros((X_test.shape[0], len(skf)))

        for i, (train_index, cv_index) in enumerate(skf):
            print 'Fold [%s]' % (i)

            X_train_f = X_train[train_index] #output from PCA, array returned  
            y_train_f = y_train.iloc[train_index] #iloc if using pandas 
            X_cv = X_train[cv_index]
            y_cv = y_train.iloc[cv_index]

            #Train on train part of fold
            clf.fit(X_train_f, y_train_f)

            #Predict on cv_index part of fold
            blend_train[cv_index, j] = clf.predict(X_cv)
            #N-folds, so i will have n-predictions on x_test
            blend_test_j[:,i] = clf.predict(X_test)

        #Take mean of n-predictions
        blend_test[:,j] = blend_test_j.mean(1)

    #Blending, use different classifier
    b_clfs = [svm.SVC(C=1.0, kernel='linear'),
              RandomForestClassifier(n_estimators = 100),
    b_clfs_name = ['SVC','RandomForestClassifier','KNeighborsClassifier']

    b_pred = np.zeros((blend_test.shape[0], len(b_clfs)))    
    for y, b_clf in enumerate(b_clfs):
        print 'Blending classifier: %s' % b_clfs_name[y]

        b_clf.fit(blend_train, y_train)
        b_pred[:,y] = b_clf.predict(blend_test)

        print 'Accuracy = %s' % (metrics.accuracy_score(y_test, b_pred[:,y]))
        print 'Precision = %s' % (metrics.precision_score(y_test, b_pred[:,y]))
        print 'Recall = %s' % (metrics.recall_score(y_test, b_pred[:,y]))

    #Final Prediction
    print '---- Final score ----'
    #b_pred_m = np.round_(np.mean(b_pred, axis=1)) #mean for each row
    b_pred_m = np.round_(np.ma.average(b_pred, axis=1,)) #weights=[0,0,0] #mean for each row 

    print 'Accuracy = %s' % (metrics.accuracy_score(y_test, b_pred_m))
    print 'Precision = %s' % (metrics.precision_score(y_test, b_pred_m))
    print 'Recall = %s' % (metrics.recall_score(y_test, b_pred_m))
  • 1
    $\begingroup$ RandomForrest is already an ensemble! $\endgroup$ – kjetil b halvorsen Apr 22 '16 at 7:08
  • 1
    $\begingroup$ I know, but still I would expect that group of models (normal and ensembled) will beat single one ensembled. $\endgroup$ – HonzaB Apr 22 '16 at 7:15

There are several approaches that you might consider taking.

Firstly, it might be worth seeing if you can weight the contribution of each model to the final ensemble by some metric based on how well they perform on your validation set.

Secondly, it might be a good idea to look at the output of each model and calculate how well correlated they are - if you have very highly correlated results then it might be better to drop some of those from the ensemble and instead combine the least correlated results (the hypothesis being that the correct guesses should be correlated and the errors uncorrelated).

You may have seen it already but there is some excellent advice about this in the context of a Kaggle competition here.

  • $\begingroup$ Yes, I was thinking the same - weight them based on their performance in CV. And for the second, i should take models which are as much different as possible right?. Thanks, I will try both. Yes, I read the article many many times ;). $\endgroup$ – HonzaB Apr 29 '16 at 6:22
  • $\begingroup$ Yes, that is right - find the models that are the least correlated with each other and see how that performs. $\endgroup$ – anthr Apr 29 '16 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.