1
$\begingroup$

I am looking for a variable selection method for linear regression. I have 25 correlated independent variables and one dependent variable that is an aggregated score of a Likert scale. I also have 90 samples. I tried LASSO with AIC and BIC but the suggested alpha from a 5-fold cross validation doesn't reduce the regressors to a number that is not overfitting my model. Any suggestions? PS I did the analysis in python but I am not proficient with it.

$\endgroup$
  • 1
    $\begingroup$ How do you know you are overfitting? AIC and BIC should not overfit. Or if you do not like the answer by AIC and BIC, try optimizing with respect to some measure of forecast error, e.g. mean squared error, when you cross validate your model. $\endgroup$ – Richard Hardy Mar 21 '17 at 7:34
1
$\begingroup$

Create a validation set, separate from your test set etc. Add variables, one at a time, eg using LARS path, which lets you do this easily, and quickly, until validation shows that validation loss is not going down and/or is increasing.

$\endgroup$
  • $\begingroup$ I did use the LARS path but the issue I get is that it works fine whenever I do the Training-Testing and decreases the predictors down to 4 or 5 but whenever I use that alpha to my whole sample it decreases them down to 18 which overfits my model. I think the issue is that I have a small sample but I don't know what to do. $\endgroup$ – Da Kor Mar 21 '17 at 13:47
  • 1
    $\begingroup$ What I mean is, you can manually choose how many features to use, according to your validation set. Add the features in the sequence indicated by LARS, but the number of features to add, you choose, based on your validation set results. $\endgroup$ – Hugh Perkins Mar 21 '17 at 23:29
  • $\begingroup$ You mean I can choose the number of the regressors? This is what I did previously: scikit-Lasso model selection: Cross-Validation / AIC / BIC Do you know of any resources to help me do that? $\endgroup$ – Da Kor Mar 22 '17 at 11:51
  • 1
    $\begingroup$ I'm not sure; I coded LARS myself ... github.com/hughperkins/selfstudy-LARS/blob/master/… (quite badly; there was a bunch of linear algebra I had yet to learn when I wrote this...) $\endgroup$ – Hugh Perkins Mar 22 '17 at 13:54
  • $\begingroup$ (maybe max_iter is what you need? But, you can just get back all the coefficients/features, and then walk along it, until the validation loss stops decreasing?) $\endgroup$ – Hugh Perkins Mar 22 '17 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.