I have a few very large and quite "dirty" (survey) datasets. Primarily, there are lots of NA's. These NA's are mostly the result of different questions being asked in different waves. It is perfectly possible that a question present in the dataset has not been asked more often than it has (it is for example asked in the first wave but not after, or in the last wave, but not before).

Because the datasets are so large I would like to use LASSO, to aid me with variable selection.

If I would however only deal with complete cases (as LASSO requires complete cases) I would not have any cases left (or probably have a huge bias).

Because of the amount of NA's I'm not sure imputation with 'mice' will be a solution (but please comment on this).

I was wondering if there are smart ways of subsetting my dataset in R. For example cutting my variables in parts. Although I have doubts whether the LASSO is then still valid.

Are there any other possibilities to deal with NA's for LASSO?

  • $\begingroup$ Could you give some more information on missingness? What fraction is missing? Is the fraction the same for all the variables? Patterns of missingness? If you cannot answer this Qs, then you should start with studying the patterns if missingness in your data ... and present to us some of that. But probably, your best bet is some method of imputation. $\endgroup$ Aug 27, 2018 at 16:49
  • $\begingroup$ I have elaborated a bit on the missing data in the original post. $\endgroup$
    – Tom
    Aug 28, 2018 at 11:31


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.