I have predictions from 3 binary classifiers (SVM, RF and NN) and would like to combine them in some way.
I'm aware of the notion of ensemble learning methods but I was wondering if it would be valid to do something as simple as majority voting on the three predictions.
E.g.:
Observation SVM RF NN Overall
One 1 1 0 1
Two 0 0 0 0
Three 0 0 1 0
Four 0 1 1 1
Would this be a reasonable and valid approach?
Thank you.
This is a perfectly valid method. The method that give the best prediction score will be considered the one to use.
However you might want to add more detail to your ensemble model. For instance RF and NN would outputs probabilities. So you could combine the probabilities to compute the majority vote instead of 0-1. SVM do not output probabilities so you could leave it as 0-1 or there is actually a way to compute probabilities on top SVM see https://www.researchgate.net/post/Can_we_assign_probability_to_SVM_results_instead_of_a_binary_output2
Another common thing to do, is to feed the output of your 3 classifiers to a logistic regression, so they all have their own weights. In a sense, it is a majority vote with "optimized weights".
If you want more details and look into various techniques I would recommend reading this article: http://mlwave.com/kaggle-ensembling-guide/
