I am using Weka to implement stacking with k-fold cross validation. As I understand, we first divide our dataset in to k folds, then we use k-1 folds for training and 1 fold for testing. This procedure is repeated k times.
Suppose we have 3 classifiers in level 0 of the stacking. After training classifiers in level 0, every time that they make predictions on a test set, we save their predictions in separate columns to make a new training dataset to train the level 1 meta classifier in order to learn their performance.
But I don't understand what will happen next. How can we create a test set for the level 1 meta classifier? Do we have to split a part of our original dataset as a test set before doing k-fold cross validation? If not, what we can do?
Extra Explanation has been added about Stacking:
Stacked generalization or stacking for short, is a 2-level combination method. In the first level we have multiple base classifiers that can be homogeneous or heterogeneous; in the second level we have a new concept called a meta-learner. Without k-fold cross validation its procedure is like this:
Suppose we have divided our data set in to 3 groups: 1) training set, 2) validation set, 3) test set. We first train our base level classifiers and let them make predictions on the validation set. Then we save their predictions (the class labels) in separate columns (or we stack them up) for creating a new training dataset for our meta-learner: the features of the new data set will be the predictions of the base classifiers plus an extra column that shows the actual class of each corresponding instance. The meta-learner is trained by comparing the predicted class for each instance with its actual class. Once trained, it can understand which classifiers had better performance and learn to trust them.
In this case, we had a test set from the beginning and now its time to use the test set to make final predictions by the meta learner. The test set first had been given to the base level classifiers, they made their predictions on each of the test instances, just like what had been done on the validation set. Their predictions on each of the test instances will be saved in separate columns to make the new test set for the meta-learner, as its the test set we don't have actual class column. After giving the test set to the meta-learner, it's time for the meta-learner to combine the outputs/predictions of the base learners to make the final prediction.
What made me confused is that in the above scenario we had 3 sets (training, validation and testing sets), and everything was understandable about how to create a new training set for the meta learner, or about how to make a new test set for the meta learner. But when using k-fold cross validation, have only 2 sets (train, test). I can't understand how to make the new test set at the final to get the final prediction from the meta-learner.