I'm trying the stacking method to see if it improves my results, but before using some R package, I decided to code it by myself. Here's a pseudocode of what I'm doing:
train.all = getTrain()
# separate 20% of data to test the stacked model
test.meta.idx = sample(nrow(train.all), floor(nrow(train.all)*0.2))
test.meta = train.all[test.meta.idx, ]
# remove these from train.all
train.all = train.all[-test.meta.idx, ]
# generate folds for cross-validation
k = 10
folds = generateFolds(k)
# dataset to store base learners predictions
train.meta = data.frame()
for (i in 1:k) {
train.idx = folds[[1]]$train
test.idx = folds[[i]]$test
train = train.all[train.idx, ]
test = train.all[test.idx, ]
# train models
model1 = fitmodel1(formula, train)
model2 = fitmodel2(formula, train)
model3 = fitmodel3(formula, train)
# get model outputs
y1 = predict(model1, test)
y2 = predict(model2, test)
y3 = predict(model3, test)
y.obs = test$y
# append to meta train.meta
train.meta = rbind(train.meta, c(y.obs, y1, y2, y3))
}
Now I can use train.meta to fit a different model, which will give the final result based on inputs from model1, model2 and model3 predictions. But, how do I test it? For each fold a fit a different model1, model2 and model3, so I will have 10 different model1, model2 , and model3.
- Should i re-refit the base learners using the entire training data?
- Is it ok to train the meta-model using the fitted base-learner values?
Thanks for any advice!