Why does Machine-Learning-Augmented Logistic Regression perform better? I have a need to build binary classifiers. For institutional reasons I am constrained to using only logistic models. The continuous and integer independent variables are typically binned in a manner that produces monotonic Weights of Evidence vs the binary target. A typical dataset will be segmented into 5-15 groups, with each group being modeled by its own LR equation.
It turns out that when the binary target is replaced with a fractional probability prediction from an XGBoost model, that the resulting LR equations have meaningfully better out-of-sample performance, than if they'd been estimated on the true binary outcome, holding equation specification the same. Can anyone explain why this result might be the case? For reference, the logistic models are typically estimated on 15-100k observations, with 6-10 independent variables. The improvement over baseline LR performance is loosely associated with sample size, smaller datasets get the biggest boost, but all models are improved. 
 A: Here's what I think is producing the pattern you see. 
Models are approximations of $\Pr(Y=1|X) = E[Y|X]$ using data $Y, X$. Compared to unregularised logistic regression, machine learning models typically do a much better job of that, judged by mean squared error, log loss etc, when $X$ is high dimensional because they are not as likely to pick up spurious relationships between $Y$ and the elements of $X$.
Suppose we could train the logit model using $E[Y|X], X$ instead of $Y, X$. Clearly, that would work much better because unlike $Y$, $E[Y|X]$ only varies when $X$ varies, which is exactly what the model is trying to figure out. Replacing $Y$ with $E[Y|X]$ gets rid of the variation in $Y$ that is unpredictable using variation in $X$.
When we replace $E[Y|X]$ by a good estimate of it coming out of a regularised model, $\widehat{E[Y|X]}$, that's still an advantage for the same reason. However in the limit as the sample size grows, keeping the dimensionality of $X$ fixed, the advantage vanishes. Logit trained on $Y,X$, logit trained on $E[Y|X], X$ and logit trained on $\widehat{E[Y|X]}, X$ all converge to the same function $f(x)$: The best approximation of $E[Y|X]$ within the class of functions logit can represent.
Effectively, machine-learning-augmented logistic regression is regularised logistic regression. You can verify this with simulations along the following lines:


*

*Simulate data eg $Y_i = 1\{ X_i \beta + u > 0\}$ with $u \sim N(0, 1)$. Make $X$ (dimension $K$) high dimensional relative to $N$, make only one element of $\beta$ have a large absolute value.

*Fit a logit model to the data. 

*Fit regularised model that recognises only one element of $X$ is really important, shrinking others to zero.

*Fit a logit model, replacing $y$ with $\hat{y}$

*Compare out of sample predictions for $y$
The model with replaced outcome does much better out of sample, but the advantage goes away as $\frac{K}{N}$ goes to zero.
Edit: Rereading your post, you seem to have a huge amount of data for only a handful of predictors, so you wouldn't think regularisation matters. If you follow Matthew Drury's suggestion and check if it still happens with regularised logistic regression that will be a test of this explanation. I also thought about whether replacing $y$ with $\hat{y}$ might boost the logistic model even when variance is no problem but bias is, but I couldn't come up with a reason why it would. 
