I want to experiment with a multi-level machine learning structure. Here is a conceptual plot:
On the first level are multiple learning algorithms that provide input for a second algorithm (the 'output layer').
My question is:
How do I generate the input for the second level during training?
I came up with two options so far:
I use a leave-k-out mechanism, such that I have k (unseen data) predictions from a trained model (using n-k samples) in the first layer. Repeat until each sample was left out once. Finally, I forward the corresponding predictions to the second layer.
Train the data and forward predictions from the (cross-validated) trained model. All predictions were seen before but it is the final function that is later used for the "overall unseen" data.
From intuition I prefer 1. because I want generalization of "overall unseen" data. I also want that the second layer "learns" errors in the first layer. I think option 1) better handles this. But I am not sure - I couldn't find any useful source (please provide if you know one).