I am looking for a variable selection method for linear regression. I have 25 correlated independent variables and one dependent variable that is an aggregated score of a Likert scale. I also have 90 samples. I tried LASSO with AIC and BIC but the suggested alpha from a 5-fold cross validation doesn't reduce the regressors to a number that is not overfitting my model. Any suggestions? PS I did the analysis in python but I am not proficient with it.
LASSO with cross validation doesn't reduce the regressors to a number that is not overfitting my model. What can I do?
1$\begingroup$ How do you know you are overfitting? AIC and BIC should not overfit. Or if you do not like the answer by AIC and BIC, try optimizing with respect to some measure of forecast error, e.g. mean squared error, when you cross validate your model. $\endgroup$– Richard HardyMar 21, 2017 at 7:34
Create a validation set, separate from your test set etc. Add variables, one at a time, eg using LARS path, which lets you do this easily, and quickly, until validation shows that validation loss is not going down and/or is increasing.
$\begingroup$ I did use the LARS path but the issue I get is that it works fine whenever I do the Training-Testing and decreases the predictors down to 4 or 5 but whenever I use that alpha to my whole sample it decreases them down to 18 which overfits my model. I think the issue is that I have a small sample but I don't know what to do. $\endgroup$– DanaiMar 21, 2017 at 13:47
1$\begingroup$ What I mean is, you can manually choose how many features to use, according to your validation set. Add the features in the sequence indicated by LARS, but the number of features to add, you choose, based on your validation set results. $\endgroup$ Mar 21, 2017 at 23:29
$\begingroup$ You mean I can choose the number of the regressors? This is what I did previously: scikit-Lasso model selection: Cross-Validation / AIC / BIC Do you know of any resources to help me do that? $\endgroup$– DanaiMar 22, 2017 at 11:51
1$\begingroup$ I'm not sure; I coded LARS myself ... github.com/hughperkins/selfstudy-LARS/blob/master/… (quite badly; there was a bunch of linear algebra I had yet to learn when I wrote this...) $\endgroup$ Mar 22, 2017 at 13:54
max_iteris what you need? But, you can just get back all the coefficients/features, and then walk along it, until the validation loss stops decreasing?) $\endgroup$ Mar 22, 2017 at 13:57