# Ensemble of models with different feature spaces

BACKGROUND

I have data in which the dependent variable is binary with a highly-skewed distribution: <1% records are 1 (doers), >99% records are 0 (non-doers). I'm using logistic regression to predict the probability that new records are doers.

To handle this rare-event situation, I made multiple samples of non-doers that are size-matched to the number of doers (e.g., sample 1 has the 100 doers and 100 non-doers, sample 2 has the 100 doers and a different set of 100 non-doers, etc.).

QUESTION

If I fit a logistic regression to each sample, how do I make an ensemble model to assign probabilities to new records? Samples have different observations and perform their own feature selection, so they have different feature spaces, which precludes averaging feature coefficients.

Do you have any suggestions for how I can build an ensemble model to take into account the models from all of my samples to compute probabilities?

• It sounds like you want to solve problems induced by two rather uncommon steps (splitting and size matching). Thus the question: why not running a single logistic regresdion on the full sample? – Michael M Apr 9 '14 at 15:30
• Thanks for your question, Michael. I cannot run a single logistic regression on the full sample because it is too large to hold in memory and it suffers from the class imbalance I discuss above. The method of undersampling to account for class imbalance is not uncommon given this situation. – Gyan Veda Apr 9 '14 at 15:52

You need a hold-out dataset that is representative of the actual mix of 0s and 1s. Use each model in the ensemble to predict probabilities for the holdout dataset. Then fit a meta-model, where the inputs are the predicted probabilities and the output is 0 or 1.

Something like a simple logistic regression might work, but I've usually had better luck in this situation with generalized additive models.

Use the predicted probabilities from your meta-model.

• Zach, thanks for your answer! I like the meta-model idea. I was thinking of averaging the probabilities across models, which is equivalent to giving them all equal weights. But the meta-model gives them weights according to their predictive power, which is more sophisticated. One thing I don't understand is why I need the hold-out sample to be representative of the actual mix of 0s and 1s. I'm already applying a correction to account for this in the intercept of my models per Tomz, King, and Zeng (2003). – Gyan Veda Apr 9 '14 at 16:07
• I like this approach too! This should naturally weight more accurate models over the less accurate ones. – Arun Jose Apr 9 '14 at 17:30
• @user2932774 if you like my answer, please feel free to upvote or accept it! And yes, corrected your intercept term should work too. – Zach Apr 9 '14 at 19:03

One approach that might work would be to use a cutoff to determine 1 or 0 for each model. Then using your ensemble, for each record calculate the sum of total 1s for each record.

Wherever you have general agreement across models you would have greater certainty.

Set a minimum agreement criteria for your final classification.

• Arun, thank you for your suggestion! If I were interested in probability estimates instead of classification scores, then do you think I could do the same thing with probabilities instead of 1s and 0s? – Gyan Veda Apr 9 '14 at 13:35

IMO this is not a classification problem, because you have much more negative example than positive, so why not let machine concentrate on "negative" learning instead of learning both.

Maybe you can try some anomaly detection technique to test if the new data "normal" (with many negative examples, machine can learn "normal" quite good), if not then it's anomaly, it doesn't even need to learn how does "anomaly" look like.