Can I use predictions from LOESS as a form of predictor transformation in logistic regression?

Question

I am wondering if I can use the predictions from LOESS as a form of predictor transformation in logistic regression? For example, if one of the predictors is X, then can I use predict(loess(Y~X)) as a transformation of X and use it in the logistic regression?

The variable X is a ratio variable which is highly concentrated at its mean while having a handful of outliers. In order to increase the predictive power of X (i.e. finding linearity between X and Y and fulfilling the business requirements), I am thinking of some transformations which can improve the linearity relationship between X and logodds.

My concern is that this kind of transformation may cause information leakage as I am using the the response variable Y to guide the transformation. However, does the popular WOE transformation have the similar concern?

Or, could there be other transformations that I can try?

Thanks.

Answer 1

This kind of transformation can cause information leakage (just think of the case where you smooth do little that the loess curve goes thorough every single point), but may also be helpful for predictive performance (it's probably a bit of a mess when you are primarily interested in inference - splines are probably better understood in that context). It's somewhat related to the popular idea of target encodings (encoding a categorical variable by the mean outcome - with various ideas like shrinking towards the mean to help avoid overfitting/target leakage), which can be very helpful for predictive performance when there's many categories.

I would first consider whether suitable splines could do the job, and of you really want to do this, a lot will depend on good regularisation (i.e. a good choice of the smoothing parameter), for which a good cross validation or other validation set up will be critical (in particular this really needs to test what happens when you apply this to new data - i.e. curve fitted to training data of each CV fold split evaluated on validation part of that fold split). There's also approaches for creating the curve feature from (within each training fold) 4/5 of the training data for the other 1/5 of the data (repeated 5 times for each 1/5 - so, if you cross validate that's 5 times per fold), which help a lot with target leakage for target encodings and might help here, too.