I have a need to build binary classifiers. For institutional reasons I am constrained to using only logistic models. The continuous and integer independent variables are typically binned in a manner that produces monotonic Weights of Evidence vs the binary target. A typical dataset will be segmented into 5-15 groups, with each group being modeled by its own LR equation.

It turns out that when the binary target is replaced with a fractional probability prediction from an XGBoost model, that the resulting LR equations have meaningfully better out-of-sample performance, than if they'd been estimated on the true binary outcome, holding equation specification the same. Can anyone explain why this result might be the case? For reference, the logistic models are typically estimated on 15-100k observations, with 6-10 independent variables. The improvement over baseline LR performance is loosely associated with sample size, smaller datasets get the biggest boost, but all models are improved.

  • $\begingroup$ Why is the target outcome replaced? $\endgroup$ – Repmat Sep 18 '19 at 19:32
  • $\begingroup$ Is the prediction from that XGBoost model a probability or a binary outcome as well? $\endgroup$ – CloseToC Sep 18 '19 at 19:49
  • $\begingroup$ question text updated, the XGBoost predictions are fractional predictions, probabilities $\endgroup$ – JPErwin Sep 18 '19 at 21:09
  • $\begingroup$ the target outcome is replaced because "when the binary target is replaced with a fractional probability prediction from an XGBoost model, that the resulting LR equations have meaningfully better out-of-sample performance" $\endgroup$ – JPErwin Sep 18 '19 at 21:10
  • $\begingroup$ Is this still true if you use regularized logistic regression? $\endgroup$ – Matthew Drury Sep 18 '19 at 21:50

Here's what I think is producing the pattern you see.

Models are approximations of $\Pr(Y=1|X) = E[Y|X]$ using data $Y, X$. Compared to unregularised logistic regression, machine learning models typically do a much better job of that, judged by mean squared error, log loss etc, when $X$ is high dimensional because they are not as likely to pick up spurious relationships between $Y$ and the elements of $X$.

Suppose we could train the logit model using $E[Y|X], X$ instead of $Y, X$. Clearly, that would work much better because unlike $Y$, $E[Y|X]$ only varies when $X$ varies, which is exactly what the model is trying to figure out. Replacing $Y$ with $E[Y|X]$ gets rid of the variation in $Y$ that is unpredictable using variation in $X$.

When we replace $E[Y|X]$ by a good estimate of it coming out of a regularised model, $\widehat{E[Y|X]}$, that's still an advantage for the same reason. However in the limit as the sample size grows, keeping the dimensionality of $X$ fixed, the advantage vanishes. Logit trained on $Y,X$, logit trained on $E[Y|X], X$ and logit trained on $\widehat{E[Y|X]}, X$ all converge to the same function $f(x)$: The best approximation of $E[Y|X]$ within the class of functions logit can represent.

Effectively, machine-learning-augmented logistic regression is regularised logistic regression. You can verify this with simulations along the following lines:

  1. Simulate data eg $Y_i = 1\{ X_i \beta + u > 0\}$ with $u \sim N(0, 1)$. Make $X$ (dimension $K$) high dimensional relative to $N$, make only one element of $\beta$ have a large absolute value.

  2. Fit a logit model to the data.

  3. Fit regularised model that recognises only one element of $X$ is really important, shrinking others to zero.

  4. Fit a logit model, replacing $y$ with $\hat{y}$

  5. Compare out of sample predictions for $y$

The model with replaced outcome does much better out of sample, but the advantage goes away as $\frac{K}{N}$ goes to zero.

Edit: Rereading your post, you seem to have a huge amount of data for only a handful of predictors, so you wouldn't think regularisation matters. If you follow Matthew Drury's suggestion and check if it still happens with regularised logistic regression that will be a test of this explanation. I also thought about whether replacing $y$ with $\hat{y}$ might boost the logistic model even when variance is no problem but bias is, but I couldn't come up with a reason why it would.

| cite | improve this answer | |
  • $\begingroup$ This makes sense to me. I do only have a few "variables" in the logistic regression, but I should have highlighted that, because the variables are binned, and then treated as dummies/one-hots there are 128-156 parameters to estimate. Looking at the results, I see the rank ordering of the performance improvement for three models is predicted by the number of observations per parameter to estimate. $\endgroup$ – JPErwin Sep 20 '19 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.