I have generated 42 features from the existing dataset for a prediction task. All these features are significantly correlated with the target variable (ranging from .25 to .05). For the dimension reduction I used PCA. However, when I run the linear regression model I obtain a very low r2 value (~0.057). At this point I don't know what to do and how to proceed. I am thinking to generate more sparse feature set to improve the prediction. Any recommendations? I appreciate any help.

  • $\begingroup$ Your $R^2$ value is over the training set? What is the number of samples? Two things to look at might be 1) the distribution of the residuals (e.g. is it "nice and Gaussian"?), and 2) put the target into the PCA (perhaps standardizing the variables first). $\endgroup$ – GeoMatt22 Feb 14 '17 at 23:47
  • $\begingroup$ PCA is not a good dimension reduction method in the context of regression, as it ignores the relationship between your features and the response. You should instead use a regularization procedure like ridge, and combine with cross validation. $\endgroup$ – Matthew Drury Feb 14 '17 at 23:52
  • $\begingroup$ @MatthewDrury thanks for your answer! I wonder if you have any recommended library/tool/software to do a regularization procedure like ridge? $\endgroup$ – renakre Feb 15 '17 at 6:35
  • $\begingroup$ @GeoMatt22 it is over the test set. I have 650 samples (30% testing). $\endgroup$ – renakre Feb 15 '17 at 6:37
  • $\begingroup$ What is the train-set $R^2$? In other words, do your PCA-reduced features over- or under-fit? (42 is the number of raw features or reduced features?) Whatever package you are currently using should be able to do ridge regression (if there is no built in routine, see here). Note that ridge is to reduce over-fitting, so depending on your answer to my first question, you would apply it to either the raw or reduced features. $\endgroup$ – GeoMatt22 Feb 15 '17 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.