I'm working on a classification model that will predict whether a sales opportunity will end up 'won' or 'lost', given various attributes of the opportunity. I've been using my training data to build several models, including a random forest model, logistic regression, a decision tree model, ada boost model, and support vector machine. Each of these models outputs a probability which is used to assign a 'won' or 'lost' label to the unknown opportunities.
I've been using these probabilities, rather than the binary 'won' or 'lost', to compute an expected value of the opportunity based on its value if it ends up being 'won'.
My question is, how should I combine the estimates from these various models to get an overall estimate (presumably better than any one individually) to compute the expected value from? The simplest is the arithmetic mean, which I've tried.
I'm also considering splitting my training data and training all the models on say 70% of the data. Then, use the models to generate probabilities on the remaining 30%, and then train a final logistic regression on that 30% using the estimated probabilities from the original models as inputs to the logistic regression. Then, rather than using the arithmetic mean of the estimates on new opportunities, I'd perform two-stages, first getting estimates from the models trained on the 70%, then getting a single final estimate by using those as input to the logistic model.
This is new territory for me, so any advice, new ideas, or suggested readings are highly appreciated.