I am trying to do model selection on some candidate predictors using LASSO with a continuous outcome. The goal is to select the optimal model with the best prediction performance, which usually can be done by K-fold cross validation after obtaining a solution path of the tuning parameters from LASSO. The issue here is that the data are from a complex multi-stage survey design (NHANES), with cluster sampling and stratification. The estimation part is not hard since glmnet in R can take sampling weights. But the cross validation part is less clear to me since observations now are not i.i.d anymore, and how can the procedure account for sampling weights representing a finite population?

So my questions are:

1) How to carry out K-fold cross validation with complex survey data to select the optimal tuning parameter? More specifically, how to appropriately partition the sample data into training and validation sets? And how to define the estimate of prediction error?

2) Is there an alternative way to select the optimal tuning parameter?

  • $\begingroup$ Maybe resampling (e.g. bootstrap) would be a more appropriate procedure instead of k fold cv? $\endgroup$
    – g3o2
    Nov 29 '17 at 0:27
  • $\begingroup$ Lumley warns that "The theory for the bootstrap was developed only for the situation of equal-probability sampling within each stratum, it is not clear how well it would work with arbitrary probabilities." (28) NHANES data don't appear to have equal-probability sampling within strata. $\endgroup$
    – Dan Hicks
    Nov 30 '17 at 20:38
  • $\begingroup$ would be interesting to know this little detail from the OP. Don't see what would be so complex about multistage cluster and stratified sampling... $\endgroup$
    – g3o2
    Dec 3 '17 at 17:52
  • $\begingroup$ @Dan Hicks: I don't think it really matters, when resampling, you replicate multiple instances of exactly the same sample plan. $\endgroup$
    – g3o2
    Dec 3 '17 at 22:44
  • $\begingroup$ Using something like the methods described here? amstat.tandfonline.com/doi/pdf/10.1080/01621459.1988.10478591 (Or something more recent?) If you want to write up that idea with a little more detail as an answer, I'll give you the bounty. $\endgroup$
    – Dan Hicks
    Dec 4 '17 at 2:14

I don't have a detailed answer, just some pointers to work I've been meaning to read:

You could take a look at McConville (2011) on complex-survey LASSO, to be sure your use of LASSO is appropriate for your data. But maybe it's not a big deal if you're doing LASSO only for variable selection, then fitting something else to the remaining variables.

For cross-validation with complex survey data (though not LASSO), McConville also cites Opsomer & Miller (2005) and You (2009). But their methods seems to use leave-one-out CV, not K-fold.

Leave-one-out should be simpler to implement with complex surveys---there's less concern about how to partition the data appropriately. (On the other hand, it can take longer to run than K-fold. And if your goal is model selection, it's known that leave-one-out can be worse than K-fold for large samples.)


EDIT by OP: Not applicable to complex survey data.

The cv.glmet function could help you to perform the cross validation required. The lambda.min value is the value of λ where the CV error is minimal. The lambda.1se represents the value of λ in the search that was simpler than the best model (lambda.min), but which has error within 1 standard error of the best model.

  1. Choose a grid of values you could choose from for both alpha and lambda

grid <- expand.grid(.alpha = (1:10) * 0.1, .lambda = (1:10) * 0.1)

  1. Setup the control parameters of your model. The below train control does repeatedcv for 10 iterations. Go over the methods available and pick the one that would fit your current scenario.

cv.glmmod <-cv.glmnet(xTrain,y=yTrain,alpha=grid$.alpha, > = T,lambda = grid$.lambda)

The lambda.min value could be accessed from the model itself as shown below.


  • 1
    $\begingroup$ I know how to do cross validation for i.i.d data using glmnet. I was asking about correlated complex survey data. $\endgroup$
    – aenima
    May 10 '16 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.