2
$\begingroup$

My situation:

  • small sample size: 116
  • binary outcome variable
  • long list of explanatory variables: 50
  • explanatory variables did not come from the top of my head; their choice was based on the literature.

Following a suggestion to a previous question of mine, I have run LASSO (using R's glmnet package) in order to select the subset of exaplanatory variables that best explain variations in my binary outcome variable.

I have noticed that I get very different values of lambda.min through k-folds cross-validation (cv.glmnet command) according the value I attribute to k. I have tried the default (10) and 5. Which would be the most appropriate value for k, considering my sample size?

In my specific case, is it necessary to repeat cross-validation, say 100 times, in order to reduce randomness and allow averaging the error curves, as is suggested in this post? If so: I have tried the code suggested in that post, but got error messages, could anyone suggest a better code?

UPDATE1: I have managed to use the foldid option in cv.glmnet, as suggested in the comments below, by organizing my x-matrix in a way that all the 32 observations belonging to one of my outcome classes appears in lines 1-32 and by using the folowing code: foldid=c(sample(rep(seq(10),length=32),sample(rep(seq(10),length=84)). However, when I ran cv.glmnet, only one of the levels of a categorical variable with four levels was included in the model. So following a suggestion to a previous question of mine, I tried to run group-lasso using R's gglasso package. And now I am facing this issue.

$\endgroup$
2
$\begingroup$

10-fold cross-validation often is considered as the gold standard because of the compromise between bias and variance. If I understand correctly (because statistics and machine learning is not my main topic), if you go to larger number of folds, your error estimate will greatly depend on your data. As a consequence, the error estimate will have high variance and low bias.

I would say, if you know that another set of samples would show approximately the same values as you have (that means you have low variance), you can use larger number of folds (even LOOCV). Otherwise, I would leave 10-fold CV.

$\endgroup$
  • 2
    $\begingroup$ A potential problem here is that, as stated by the OP in other questions, there are only about 30 "events" in these binary-outcome data. So there's a reasonable chance that, in some of the 10-fold test samples, there will be no events, perhaps limiting the ability to cross-validate. $\endgroup$ – EdM Sep 30 '14 at 16:41
  • 1
    $\begingroup$ @EdM In that case, I would advise composing folds via sampling on the outcome. Because this is a logistic regression, this will only invalidate the intercept term (which is not estimated in lasso anyway), but not the coefficients' estimates. Substitute the appropriate intercept term value when assessing performance: the ML estimate of the intercept is fixed at the ratio of outcome 1 to outcome 0, so it's basically meaningless anyway. $\endgroup$ – Sycorax Sep 30 '14 at 16:56
  • 1
    $\begingroup$ @Puzzled I'm saying that you select folds such that every fold is composed of the same number of observations from each class. This is easy enough to do: create a data frame of class 1 and sample from it. Do the same for class 0. Concatenate them, train the model, test out-of-sample performance, and repeat. $\endgroup$ – Sycorax Sep 30 '14 at 17:11
  • 2
    $\begingroup$ Yes, by "events" I mean the smaller of the 2 outcome categories. I like the solution proposed by @user777, to sample separately from each of the outcome classes. Choose your folds by sampling before you run your LASSO, and use the foldid argument to let cv.glmnet know which fold each case belongs to. $\endgroup$ – EdM Sep 30 '14 at 20:18
  • 1
    $\begingroup$ @Puzzled There is a good book named 'Introduction to statistical learning with applications in R' (www-bcf.usc.edu/~gareth/ISL). It is a very easygoing book and I liked it a lot. Have a look on the chapter about cross-validation. Moreover, lasso is described quite vividly. $\endgroup$ – Kirill Oct 1 '14 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.