1
$\begingroup$

I am looking for a variable selection method for linear regression. I have 25 correlated independent variables and one dependent variable that is an aggregated score of a Likert scale. I also have 90 samples. I tried LASSO with AIC and BIC but the suggested alpha from a 5-fold cross validation doesn't reduce the regressors to a number that is not overfitting my model. Any suggestions? PS I did the analysis in python but I am not proficient with it.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ How do you know you are overfitting? AIC and BIC should not overfit. Or if you do not like the answer by AIC and BIC, try optimizing with respect to some measure of forecast error, e.g. mean squared error, when you cross validate your model. $\endgroup$
    – Richard Hardy
    Mar 21, 2017 at 7:34

1 Answer 1

Reset to default
1
$\begingroup$

Create a validation set, separate from your test set etc. Add variables, one at a time, eg using LARS path, which lets you do this easily, and quickly, until validation shows that validation loss is not going down and/or is increasing.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I did use the LARS path but the issue I get is that it works fine whenever I do the Training-Testing and decreases the predictors down to 4 or 5 but whenever I use that alpha to my whole sample it decreases them down to 18 which overfits my model. I think the issue is that I have a small sample but I don't know what to do. $\endgroup$
    – Danai
    Mar 21, 2017 at 13:47
  • 1
    $\begingroup$ What I mean is, you can manually choose how many features to use, according to your validation set. Add the features in the sequence indicated by LARS, but the number of features to add, you choose, based on your validation set results. $\endgroup$
    – Hugh Perkins
    Mar 21, 2017 at 23:29
  • $\begingroup$ You mean I can choose the number of the regressors? This is what I did previously: scikit-Lasso model selection: Cross-Validation / AIC / BIC Do you know of any resources to help me do that? $\endgroup$
    – Danai
    Mar 22, 2017 at 11:51
  • 1
    $\begingroup$ I'm not sure; I coded LARS myself ... github.com/hughperkins/selfstudy-LARS/blob/master/… (quite badly; there was a bunch of linear algebra I had yet to learn when I wrote this...) $\endgroup$
    – Hugh Perkins
    Mar 22, 2017 at 13:54
  • $\begingroup$ (maybe max_iter is what you need? But, you can just get back all the coefficients/features, and then walk along it, until the validation loss stops decreasing?) $\endgroup$
    – Hugh Perkins
    Mar 22, 2017 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.