I have a data set with more features than observations, i.e. $p>n$. Using Lasso regression with glmnet, the optimal selection of $\lambda$ from cross-validation selects $p=8$ non-zero feature coefficients. I am trying to check predictive accuracy of the resulting model. I am use $k$-fold cross validation. Incidentally this is also point where BIC is lowest. Adjusted R^2 is about 80%. I am trying to analyse residuals(r).
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ Does your data have a spatial or temporal element that's potentially causing autocorrelation in the residuals? $\endgroup$– dclMay 3, 2018 at 3:33
-
1$\begingroup$ Are you building a model for prediction? If so, issues of residual normality and p-values are not relevent. $\endgroup$– Matthew DruryMay 3, 2018 at 3:44
-
$\begingroup$ @MatthewDrury can you kindly elaborate on this? $\endgroup$– itthrillMay 3, 2018 at 4:09
-
4$\begingroup$ There's a lot of information on this site, but the sort answer is this: those assumptions are there to support certain types of analysis using a linear model. p-values are to test the hypothesis that the coefficient is zero in a correctly specified model, and the normal assumption is there to ensure that the p-values reported are correct in this circumstance. If you are not using the model to test hypothesis that the parameters themselves are zero, then both these concerns are irrelevent to you. If your goal is making predictions, you live and die by cross validation. $\endgroup$– Matthew DruryMay 3, 2018 at 4:25
-
2$\begingroup$ In particular, p-values have nothing to do with the predictive power of the variable, and in no circumstances are there to measure if the variable should or should not be included in a model. If you are not using the model to test hypotheses, they are just noise in your regression output. $\endgroup$– Matthew DruryMay 3, 2018 at 4:26
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
2
- I do not see much evidence of heteroskedasticity in your data - the residuals look reasonably uniform across fitted values.
- Lasso will handle multicollinearity by shrinking all but one correlated feature; ridge or elastic net regularized regression are generally preferred in such situations since they penalize these features equally. The glmnet vignette describes the difference:
It is known that the ridge penalty shrinks the coefficients of correlated predictors towards each other while the lasso tends to pick one of them and discard the others.
- It should be expected that the lasso and OLS fits diverge, particularly when collinearities exist; lasso is performing a feature selection (finding a parsimonious model), while OLS will fit whatever you ask of it (always producing a more complex model than the lasso).
-
$\begingroup$ based on the F test it is hetroskedasticity. Also the cyclicity in residual plot worries me $\endgroup$– itthrillMay 3, 2018 at 4:11
-
$\begingroup$ The F test performed by a summary.lm in R does not test a hypothesis of heteroskedasticity. It tests the reduction in residuals for a full model against an intercept only model; the results of the F test may be inconsistent if there is non-constant variance, but we cannot draw that inference from the F-test result. $\endgroup$– kholMay 3, 2018 at 4:26