Ensemble of models with different feature spaces BACKGROUND
I have data in which the dependent variable is binary with a highly-skewed distribution: <1% records are 1 (doers), >99% records are 0 (non-doers). I'm using logistic regression to predict the probability that new records are doers.
To handle this rare-event situation, I made multiple samples of non-doers that are size-matched to the number of doers (e.g., sample 1 has the 100 doers and 100 non-doers, sample 2 has the 100 doers and a different set of 100 non-doers, etc.).
QUESTION
If I fit a logistic regression to each sample, how do I make an ensemble model to assign probabilities to new records? Samples have different observations and perform their own feature selection, so they have different feature spaces, which precludes averaging feature coefficients.
Do you have any suggestions for how I can build an ensemble model to take into account the models from all of my samples to compute probabilities?
 A: You need a hold-out dataset that is representative of the actual mix of 0s and 1s.  Use each model in the ensemble to predict probabilities for the holdout dataset.  Then fit a meta-model, where the inputs are the predicted probabilities and the output is 0 or 1.
Something like a simple logistic regression might work, but I've usually had better luck in this situation with generalized additive models.
Use the predicted probabilities from your meta-model.
A: One approach that might work would be to use a cutoff to determine 1 or 0 for each model. Then using your ensemble, for each record calculate the sum of total 1s for each record. 
Wherever you have general agreement across models you would have greater certainty.
Set a minimum agreement criteria for your final classification.
A: IMO this is not a classification problem, because you have much more negative example than positive, so why not let machine concentrate on "negative" learning instead of learning both. 
Maybe you can try some anomaly detection technique to test if the new data "normal" (with many negative examples, machine can learn "normal" quite good), if not then it's anomaly, it doesn't even need to learn how does "anomaly" look like.
A: If you use a logistic regression ensemble learner across your sample models you will overfit to your training data. The reason being, and assuming you are doing a random subsampling or bootstrapping of your negative class, there's no reason one sample model should be chosen over the next. They should all have equal weight in your ensemble.
My suggestion would be to simply average across the responses from each of your sample models.
