1
$\begingroup$

I recently engineered a replacement for one of the most important features in my production model. This replacement makes more intuitive sense than the current version of the feature, and I was excited to see its performance.

However, I found that the feature always makes the model (a multiple linear regression) perform slightly worse.

I investigated this and found something interesting. When measuring Pearson correlation with the response, the old feature outperforms my new feature. BUT when measuring Spearman correlation with the response, my new feature absolutely crushes the old feature!

Of course, it makes sense that the linear regression model would prefer the feature with the higher Pearson correlation. But, what I'm wondering is if the much higher Spearman correlation of my new feature is a promising sign.

Does it suggest that with the right transformation of my new feature I can give it a higher Pearson correlation, too? If so, can you point me in the right general area of some transformation techniques I should try? I've tried clipping some outliers and that seemed to help the Pearson correlation, but it still lags the old feature.

Or maybe the higher Spearman correlation is giving me false hope?

$\endgroup$
1
  • 3
    $\begingroup$ Have you looked at the scatterplot? I wonder if curvilinearity might be at play here. $\endgroup$ – Patrick Malone Apr 29 '18 at 13:01
3
$\begingroup$

One thing to keep in mind is that the bivariate correlation of a response and a potential predictor may be high while the predictor might still add very little value to a regression with a number of other predictors already present. While a high Spearman correlation does indeed suggest that some monotonic transformation of the predictor will have a high Pearson correlation with the response, it doesn't necessarily imply that it will be particularly useful in the regression.

It might be - but you won't know until you try it.

$\endgroup$
1
  • $\begingroup$ Good point. In this case, however, I do not believe the problem is my new feature's correlation with the other features in the model. I believe the problem lies in its weaker linear relationship with the response in comparison with the old feature. $\endgroup$ – dotfit Apr 29 '18 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.