Model Stacking algorithm

Question

I'm trying the stacking method to see if it improves my results, but before using some R package, I decided to code it by myself. Here's a pseudocode of what I'm doing:

train.all = getTrain()

# separate 20% of data to test the stacked model
test.meta.idx = sample(nrow(train.all), floor(nrow(train.all)*0.2))
test.meta = train.all[test.meta.idx, ]

# remove these from train.all
train.all = train.all[-test.meta.idx, ]

# generate folds for cross-validation
k = 10
folds = generateFolds(k)

# dataset to store base learners predictions
train.meta = data.frame()

for (i in 1:k) {
   train.idx = folds[[1]]$train
   test.idx = folds[[i]]$test

   train = train.all[train.idx, ]
   test = train.all[test.idx, ]

   # train models
   model1 = fitmodel1(formula, train)
   model2 = fitmodel2(formula, train)
   model3 = fitmodel3(formula, train)

  # get model outputs
  y1 = predict(model1, test)
  y2 = predict(model2, test)
  y3 = predict(model3, test)

  y.obs = test$y

  # append to meta train.meta
  train.meta = rbind(train.meta, c(y.obs, y1, y2, y3))
}

Now I can use train.meta to fit a different model, which will give the final result based on inputs from model1, model2 and model3 predictions. But, how do I test it? For each fold a fit a different model1, model2 and model3, so I will have 10 different model1, model2 , and model3.

1. Should i re-refit the base learners using the entire training data?
2. Is it ok to train the meta-model using the fitted base-learner values?

Thanks for any advice!

Answer 1

If I understand your pseudo-code correctly, I don't see where the stacking model is being tested in the cross validation loop. I would expect to see something like

model4 = fitmodel4(model1, model2, model3, train)
y4 = predict(model4, test)

Similar to how the base models' hyper-parameters are being tuned using cross validation prediction error (e.g. neural network's number of nodes, regression's independent variables and potential transformations), there would also be tuning of the stacking model's hyper-parameters in the cross validation loop

As for your questions:

1. Yes, the final base learners are fit using the entire training data
2. Yes. The stacking model would use the outputs from the final base learners. Over fitting is related to model complexity and not training on more data.