Higher Spearman but lower Pearson correlation?

Question

I recently engineered a replacement for one of the most important features in my production model. This replacement makes more intuitive sense than the current version of the feature, and I was excited to see its performance.

However, I found that the feature always makes the model (a multiple linear regression) perform slightly worse.

I investigated this and found something interesting. When measuring Pearson correlation with the response, the old feature outperforms my new feature. BUT when measuring Spearman correlation with the response, my new feature absolutely crushes the old feature!

Of course, it makes sense that the linear regression model would prefer the feature with the higher Pearson correlation. But, what I'm wondering is if the much higher Spearman correlation of my new feature is a promising sign.

Does it suggest that with the right transformation of my new feature I can give it a higher Pearson correlation, too? If so, can you point me in the right general area of some transformation techniques I should try? I've tried clipping some outliers and that seemed to help the Pearson correlation, but it still lags the old feature.

Or maybe the higher Spearman correlation is giving me false hope?

Answer 1

One thing to keep in mind is that the bivariate correlation of a response and a potential predictor may be high while the predictor might still add very little value to a regression with a number of other predictors already present. While a high Spearman correlation does indeed suggest that some monotonic transformation of the predictor will have a high Pearson correlation with the response, it doesn't necessarily imply that it will be particularly useful in the regression.

It might be - but you won't know until you try it.