I'm working on a classification model that will predict whether a sales opportunity will end up 'won' or 'lost', given various attributes of the opportunity. I've been using my training data to build several models, including a random forest model, logistic regression, a decision tree model, ada boost model, and support vector machine. Each of these models outputs a probability which is used to assign a 'won' or 'lost' label to the unknown opportunities.

I've been using these probabilities, rather than the binary 'won' or 'lost', to compute an expected value of the opportunity based on its value if it ends up being 'won'.

My question is, how should I combine the estimates from these various models to get an overall estimate (presumably better than any one individually) to compute the expected value from? The simplest is the arithmetic mean, which I've tried.

I'm also considering splitting my training data and training all the models on say 70% of the data. Then, use the models to generate probabilities on the remaining 30%, and then train a final logistic regression on that 30% using the estimated probabilities from the original models as inputs to the logistic regression. Then, rather than using the arithmetic mean of the estimates on new opportunities, I'd perform two-stages, first getting estimates from the models trained on the 70%, then getting a single final estimate by using those as input to the logistic model.

This is new territory for me, so any advice, new ideas, or suggested readings are highly appreciated.


The literature calls this ensembling, which may be the word you're missing for google.

Your logistic regression idea -- which seems to me to be building vectors [ y_i, predictions of y_i from your 5 models] then simply running a regression to find the optimal weights, seems reasonable. Note you'll then need two holdout sets: one used to build the ensemble weights and another used to estimate performance. If this is too much data to not use, and you can train your models rapidly enough, perhaps try a 10 kfold.

Also, perhaps try a tweedie regression; it will deal with with a mass at zero -- ie you have something like a mixture model with a some probability of a deal being won and then a further probability over the deals won of the value.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.