I've had an idea of a training scheme for multiple machine learning models, and want to know if it makes sense or it already has a name. The idea is to train models kinda like a swarm mind (I was watching this video when thought of that).

The outline of an algorithm is as follows and seems pretty simple: for every batch of data, pick a sample of models and train each model in order with additional features that contain predictions of every previous model on that batch, take another batch and repeat.

Of course there are a lot of variations of how to train each individual model, how to use predictions as features, when to stop and so on. I just wanna know a bit more before starting experimenting with that.


Your approach sounds interesting, although I am not completely sure that I understand it completely, especially how it would be used in practice. If the later iterations are not only trained of the input but also on the output of the earlier algorithms, wouldn't you need these outputs too for later applications? If you later (after training) want to apply your model to a real set of data, wouldn't that mean that you have to keep all the old models and rerun them to generate their output?

To answer your question: You might look into Boosting or Hierarchical Boosting. That seems to be going in the direction you are think

Edit[see comments]: Similar approaches of "stacking" multiple learners have been used under the name of Hierarchical Multi-Layer Perceptron in (earlier) research into speech recognition.

  • $\begingroup$ Yes, the idea is to keep all the models, but they're aware of the predictions of other models, both while training and on real data. The order of models can be tuned and fixed, or always be a random sample. And I'm not sure that's the same as boosting (atleast in practice), as I plan to train multiple NN and decision tree algorithms together like this. $\endgroup$ – swish Sep 17 '17 at 9:37
  • $\begingroup$ You are right, boosting implies usually that the models are trained separately from each other. In your approach, the second(third, forth..) model would mainly learn to correct the errors of the previous model. I don't think there is a name for these kind of algorithms, but I think I saw something similar to what you are describing in earlier speech-recognition research. After a short search, I found a paper on "Hierarchical MLP" that seems to go in that direction (idiap.ch/~dimseng/Idiap_IIR_104-2010.pdf) $\endgroup$ – Bobipuegi Sep 18 '17 at 11:54
  • $\begingroup$ PS: I am not sure if you'll get better approaches compared to using simple "deeper" structures, but it's worth a try $\endgroup$ – Bobipuegi Sep 18 '17 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.