3
$\begingroup$

How do people use stacking or meta ensembling with cross validation in practice and in machine learning competitions like on Kaggle? Here are two approaches I've seen (but maybe neither is correct?)


Method1 (probably introduces a leak)

splits: A B C

First Layer Models

  • fit {KNN, SVM} on [A, B], predict on C -> C'
  • fit {KNN, SVM} on [B, C], predict on A -> A'
  • fit {KNN, SVM} on [C, A], predict on B -> B'

Meta Ensemble

  • fit LogReg on [A’, B’], predict on C’
  • fit LogReg on [B’, C’], predict on A’
  • fit LogReg on [C’, A’], predict on B’

Method2

splits: A B C D

First Layer Models (Fold D)

  • fit {KNN, SVM} on [A, B], predict on C -> C'
  • fit {KNN, SVM} on [B, C], predict on A -> A'
  • fit {KNN, SVM} on [C, A], predict on B -> B'
  • fit {KNN, SVM} on [A, B, C], predict on D -> D'

Meta Ensemble (Fold D)

  • fit LogReg on [A’, B’, C’], predict on D’

Repeat for folds A-C


I think method1 introduces a leak because, for example, when predicting on C' you're using the predictions of A' as features, which depend on the target values of C for the first level model fitting. On the other hand, Method 2 seems to prevent leakage, but it's kind of complex and it reduces the data used for fitting 1st level models. How are people stacking in practice?

$\endgroup$
1
$\begingroup$

I think the only way to really determine this is to experiment. I made a small one here. I split the dataset in two and trained a model and the stacking model with the same training data at the core. In the 2nd one I trained it on the other half of the data. The accuracy of the second was was higher slightly. However, this could be explained away by the additional data that model gets to see. At the end of the day I think either method will work as long as the underlying models generalize well. It will also depend on how many observations there are to play with, training time, etc.

library(caret)

data("segmentationData")

segmentationData <- segmentationData[,c(-1,-2)]

inTrain = createDataPartition(segmentationData$Class, list = FALSE, p = 0.5)

x.train <- segmentationData[inTrain,]
x.lg <- segmentationData[-inTrain,]

fit.knn <- train(Class ~ ., x.train, method = "knn")
fit.svm <- train(Class ~ ., x.train, method = "svmRadial")

## Train Logistic Regression with same training data
e.train <- data.frame(knn = predict(fit.knn, x.train), svm = predict(fit.svm, x.train), Class = x.train$Class)
fit.lgB <- train(Class ~ ., e.train, method = "glm")

## Train Logistic Regression with different training data
e.train <- data.frame(knn = predict(fit.knn, x.lg), svm = predict(fit.svm, x.lg), Class = x.lg$Class)
fit.lgB <- train(Class ~ ., e.train, method = "glm")


resamps <- resamples(list(diff = fit.lgB, same = fit.lgA))

library(lattice)
bwplot(resamps)

enter image description here

> summary(resamps)

Call:
summary.resamples(object = resamps)

Models: diff, same 
Number of resamples: 25 

Accuracy 
       Min. 1st Qu. Median   Mean 3rd Qu.   Max. NA's
diff 0.7762  0.8142 0.8262 0.8249  0.8356 0.8753    0
same 0.7865  0.8037 0.8128 0.8148  0.8255 0.8538    0

Kappa 
       Min. 1st Qu. Median   Mean 3rd Qu.   Max. NA's
diff 0.5019   0.601 0.6273 0.6214  0.6466 0.7257    0
same 0.5380   0.575 0.5917 0.5955  0.6223 0.6776    0

Perhaps use this as a template for your own experiments :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.