I am new to applying linear regression on datasets. I have experience mostly from Coursera courses and MOOCs. There are certain dilemma i am facing when I look at the feature and their correlation to the target. I would list my issues in points for clarity.

1.What should i do when i have correlated features (i have read i could use PCA to address this) but since i have small number of features, i don't want to reduce it further using PCA. And those correlated features do bring certain information on their own. Assumptions of linear regression state that feature should not be correlated among themself. What should I do to address these issues?

2.Do features that show very little correlation with the target (say .1) bring any useful information in linear regression. Should I use these uncorrelated features in my model or should I drop them?

I and using pandas and sklearn for my project.

  • $\begingroup$ 1. You can use regularization. Or use PCA, but take all principal components, not only first one or first two, if you insist on not reducing the number of predictors. 2. Explore using scatter plot to see if it's truly unrelated with the response, or if it's related not in linear way. Maybe quadratic, log linear, or whatever. If there is no relation whatsoever, you can drop it, because this predictor may cause noise rather than information to the final model. $\endgroup$ – Nuclear03020704 Aug 4 '20 at 5:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.