2
$\begingroup$

I would like to predict a binary response variable $Y_i$ using sets of predictors $\textbf{X}_{1i}, \textbf{X}_{2i}, \textbf{X}_{3i}$ for $i=1,\dots,n$. Each $\textbf{X}$ contains a few dozen predictors. For a large enough $n$, I could simply fit a standard logistic regression. However in my setting, $n$ is relatively small compared to dim($\textbf{X}$), so that it's difficult to fit such a model.

Alternatively, I have available separate datasets with $Y$ and each $\textbf{X}$ (with much larger $n$). Since I don't have a good way to combine these datasets into one large dataset, I'd like to fit three individual models and then ensemble them in some way based on my "wide but short" dataset.

Now I've got a set of 3 probabilities $p_1, p_2, p_3$ which I'd like to combine into some overall probability $p$. I've considering combining them on the following scales

  • Probability
  • Odds
  • Log-odds

The log-odds approach seems initially appealing since it will have predictors that live on the real line (seems reasonable that it could be close to linearly related with the logit of the response $Y$, but I'm not entirely sure.

I've found this paper, which seems related, but a slightly different setting, since I don't have "many forecasters".

Is there some theoretical justification for choosing one of these (or something completely different) to ensemble my three models?

$\endgroup$
  • $\begingroup$ What makes it difficult to combine the datasets? What is the exact number of candidate predictors and the counts of $Y=0$ and $Y=1$? $\endgroup$ – Frank Harrell Jun 21 at 11:12
0
$\begingroup$

Recommendation one: Cross validation to compare competing options

Try all three methods and use cross validation to see which one performs best given your error metric.

Recommendation two: Wide dataset with regularization

Instead of making three datasets and combining the results later, just make one really wide one and use l1 or l2 regularization to "solve" the dimensionality problem. Use cross validation to find your tuning parameters.

Recommendation three: Layered ensembling

You should not ensemble in-sample predictions and cross validate different ways of combining them. Instead, you should try to ensemble your cross validated predictions. This means that you do the exact same thing as in recommendation one, except instead of combining the in-sample predictions, you create cross validated predictions to ensemble.

$\endgroup$
  • $\begingroup$ Agree with recommendation 2 - Regularization techniques should be explored, especially LASSO which works "well" when number of predictors is large relative to number of observations. Besides if you have so many predictors, it is likely that some of them are not so important, and then LASSO could help you identify these less important variables. $\endgroup$ – Umka Feb 18 at 17:12
  • $\begingroup$ "Well" is questionable. The probability that lasso finds the "right" predictors is very, very low. Separate feature selection from prediction. For optimizing predictions, ridge logistic regression (or better: Bayesian models with shrinkage priors) are recommended. $\endgroup$ – Frank Harrell Jun 21 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.