I recently engineered a replacement for one of the most important features in my production model. This replacement makes more intuitive sense than the current version of the feature, and I was excited to see its performance.
However, I found that the feature always makes the model (a multiple linear regression) perform slightly worse.
I investigated this and found something interesting. When measuring Pearson correlation with the response, the old feature outperforms my new feature. BUT when measuring Spearman correlation with the response, my new feature absolutely crushes the old feature!
Of course, it makes sense that the linear regression model would prefer the feature with the higher Pearson correlation. But, what I'm wondering is if the much higher Spearman correlation of my new feature is a promising sign.
Does it suggest that with the right transformation of my new feature I can give it a higher Pearson correlation, too? If so, can you point me in the right general area of some transformation techniques I should try? I've tried clipping some outliers and that seemed to help the Pearson correlation, but it still lags the old feature.
Or maybe the higher Spearman correlation is giving me false hope?