I want to experiment with a multi-level machine learning structure. Here is a conceptual plot:


On the first level are multiple learning algorithms that provide input for a second algorithm (the 'output layer').

My question is:

How do I generate the input for the second level during training?

I came up with two options so far:

  1. I use a leave-k-out mechanism, such that I have k (unseen data) predictions from a trained model (using n-k samples) in the first layer. Repeat until each sample was left out once. Finally, I forward the corresponding predictions to the second layer.

  2. Train the data and forward predictions from the (cross-validated) trained model. All predictions were seen before but it is the final function that is later used for the "overall unseen" data.

From intuition I prefer 1. because I want generalization of "overall unseen" data. I also want that the second layer "learns" errors in the first layer. I think option 1) better handles this. But I am not sure - I couldn't find any useful source (please provide if you know one).

  • $\begingroup$ Read about ensemble learning; in special, what's called stacking. $\endgroup$ – Firebug Jan 29 '18 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.