0
$\begingroup$

I am regressing one variable on 1,500 features. I have 30,000 rows of data.

I know some of my features are correlated, but a Ridge regression where I select the ridge parameter $\lambda$ by cross validation (I use Python scikit learn) gives a very small parameter : $1e-13$. So I am doing a vanilla linear regression.

I suspect spurious regression. When I take at random only 20 features for example, my $R^2$ falls near 0, but as soon as the number of features is huge (700 for example) I have a very good $R^2$ on the test data, even if my features are correlated (many of them are moving averages of different periods).

Can I trust my $R^2$, especially that it is calculated on test data and use the model for prediction, or am I doing something wrong?

Improve this question
$\endgroup$
3
  • $\begingroup$ Why do you think something is wrong? Maybe the first half of the features happened to be more informative than the second half, and no single feature is very informative. This doesn't seem crazy. $\endgroup$
    – Danica
    Commented Oct 16, 2015 at 7:41
  • $\begingroup$ It is because I know my features are correlated. A PCA of the features reveals that the first 50 eigen values are large (between 1 and 300,000), whereas the others are very very small (near 1e-12). So I know the matrix $X^{T}X$ in the regression equation is ill-conditioned. But still, scikit learn does not find a significant ridge parameter, and my results have goor $R^2$ on test data $\endgroup$
    – volatile
    Commented Oct 16, 2015 at 7:55
  • $\begingroup$ Thanks Cagdas. My problem here is that generally a bad designed regression may result in a high $R^2$ for the training data but clearly fails for the test data. Here, I know that my regression is badly designed : correlated features, overlapping periods for mobile averages, but I still have high $R^2$ on every day of data (training on half of the day, test on the other half). Nevertheless, my parameters are not ststable from day to day. $\endgroup$
    – volatile
    Commented Oct 16, 2015 at 10:04

1 Answer 1

Reset to default
2
$\begingroup$

This situation sounds fairly reasonable to me.

Correlated features definitely result in unstable parameter estimates, as you're seeing, so don't rely on those inferences too much. But predictions can still be reasonable even if your parameter estimates are unstable: the problem is that a lot of parameters give similar predictions, so it's hard to distinguish between them, but the predictions are still fine. Assuming that your training-test split is sufficiently hygienic, that it does well on test data means that it's a pretty good predictor.

The fact that individual features don't yield good predictors doesn't mean that their combination is bad. If you're interested in reducing the multicollinearity, you could try using the principal components, or perhaps a sparse PCA to increase interpretability.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.