# How to handle with missing values in order to prepare data for feature selection with LASSO?

My situation:

• small sample size: 116
• binary outcome variable
• long list of explanatory variables: 44
• explanatory variables did not come from the top of my head; their choice was based on the literature.
• most cases in the sample and most variables have missing values.

Approach to feature selection chosen: LASSO

R's glmnet package won't let me run the glmnet routine, apparently due to the existence of missing values in my data set. There seems to be various methods for handling missing data, so I would like to know:

• Does LASSO impose any restriction in terms of the method of imputation that I can use?
• What would be the best bet for imputation method? Ideally, I need a method that I could run on SPSS (preferably) or R.

UPDATE1: It became clear from some of the answers below that I have do deal with more basic issues before considering imputation methods. I would like to add here new questions regarding that. On the the answer suggesting the coding as constant value and the creation of a new variable in order to deal with 'not applicable' values and the usage of group lasso:

• Would you say that if I use group LASSO, I would be able to use the approach suggested to continuous predictors also to categorical predictors? If so, I assume it would be equivalent to creating a new category - I am wary that this may introduce bias.
• Does anyone know if R's glmnet package supports group LASSO? If not, would anyone suggest another one that does that in combination with logistic regression? Several options mentioning group LASSO can be found in CRAN repository, any suggestions of the most appropriate for my case? Maybe SGL?

This is a follow-up on a previous question of mine (How to select a subset of variables from my original long list in order to perform logistic regression analysis?).

OBS: I am not a statistician.

• (1) The best approach to imputation depends on the proportion & pattern of missing values, the relationships between the variables, & what assumptions you're prepared to make about the reasons for missing values. (2) Any single imputation method can be used to provide input to LASSO; the difficulty's in assessing how imputation affects the results. I don't know how to combine multiple imputation with LASSO (doubtless someone does), but an informal comparison of results from different imputation runs (are the same predictors usually selected?) could still be informative. – Scortchi Jun 23 '14 at 9:13
• @Scortchi: Most of my missing values fall in the category 'not-applicable'. E.g.: In the variable 'age of adult female in the household', cases in which the adult male is a widow. Actually, I guess I need to take a step back here: should I treat values of 0 in continuous variables as missing values? E.g: 0 years of education, 0 household members between 14 and 60 years old. – Puzzled Jun 23 '14 at 23:55
• It's hard to imagine a situation in which you'd want to treat that as an unknown fact about an absent female rather than a known fact about the household. The question about zeroes is hard to understand: are you asking whether e.g. no years of education is an implausible value, or 0 might be used to code a missing value? (And then how would I know?) There's certainly no general reason to treat 0 as indicating missingness. – Scortchi Jun 24 '14 at 8:37
• @Scortchi: About the female age variable, I see what you mean. But the issue than becomes: how would I code the absent female case if not as NA? About zeroes: yes, that was exactly my question, sorry if it was not clear. I thought that the program might have some problem in handling zero values and that it might not 'understand' what I meant with it. – Puzzled Jun 24 '14 at 12:10
• You could code it as any constant value & introduce an indicator variable for presence/absence (& use group LASSO). No reason why LASSO or any other regression program should handle zero values for predictors wrongly. [Please don't take this the wrong way, but these are very basic questions, suggesting that if this is for fun you might want to start off with simpler problems, or that if it's for real you might want to consult a statistician.] – Scortchi Jun 24 '14 at 13:04

When a continuous predictor $$x$$ contains 'not applicable' values it's often useful to code it using two variables:

$$x_1=\Big{\{} \begin{array}{ll} c & \text{when x is not applicable}\\ x & \text{otherwise} \end{array} \Bigg{.}$$

where $$c$$ is a constant, &

$$x_2=\Big{\{} \begin{array}{ll} 1 & \text{when x is not applicable}\\ 0 & \text{otherwise} \end{array} \Bigg{.}$$

Suppose the linear predictor for the response is given by

$$\eta = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \ldots$$

which resolves to

$$\eta = \beta_0 + \beta_1 x_1 + \ldots$$

when $$x$$ is measured, or to

$$\eta = \beta_0 + \beta_1 c + \beta_2 + \ldots$$

when x is 'not applicable'. The choice of $$c$$ is arbitrary, & does not affect the estimates of the intercept $$\beta_0$$ or the slope $$\beta_1$$; $$\beta_2$$ describes the effect of $$x$$'s being 'not applicable' compared to when $$x=c$$.

This isn't a suitable approach when the response varies according to an unknown value of $$x$$: the variability of the 'missing' group will be inflated, & estimates of other predictors' coefficients biased owing to confounding. Better to impute missing values.

Use of LASSO introduces two problems:

1. The choice of $$c$$ affects the results as the amount of shrinkage applied depends on the magnitudes of the coefficient estimates.
2. You need to ensure that $$x_1$$ & $$x_2$$ are either both in or both out of the model selected.

You can solve both of these by using rather the group LASSO with a group comprising $$x_1$$ & $$x_2$$: the $$L_1$$-norm penalty is applied to the $$L_2$$-norm of the orthonormalized matrix $$\left[\vec{x_1}\ \vec{x_2}\right]$$. (Categorical predictors are the poster child for group LASSO—you'd just code 'not applicable' as a separate level, as often done in unpenalized regression.) See Meier et al (2008), JRSS B, 70, 1, "The group lasso for logistic regression" & grplasso.

• Does anyone know if R's glmnet package supports group LASSO? If not, would anyone suggest another one that does that in combination with logistic regression? Several options mentioning group LASSO can be found in CRAN repository, any suggestions of the most appropriate for my case? Maybe SGL? – Puzzled Jun 30 '14 at 13:03
• So, would you say that if I use group LASSO, I would be able to use the approach you suggest to continuous predictors also to categorical predictors? – Puzzled Jun 30 '14 at 22:42

Multiple Imputation is never a bad approach. You could also do Full Information Maximum Likelihood. Good review and comparison here and here.

But if you're going that route, consider using Stan to fit the ML imputation simultaneously with your regression as a single Bayesian model, since LASSO is a special case of Bayesian regression anyway.

• I had misunderstood the multiple imputation method, now I see that it would be applicable for my case. I edited my question in order to reflect this. Do you know if either SPSS or R run two options that you mentioned? – Puzzled Jun 23 '14 at 0:44
• There's an R package mi that might help you. – shadowtalker Jun 23 '14 at 1:40
• You can run Stan through R (see RStan). – Scortchi Jun 23 '14 at 9:16
• Additional multiple imputation packages for R include Amelia and mice. – Sycorax Sep 4 '14 at 14:42

The CATREG command in Statistics handles missing data with LASSO. You can exclude cases listwise or have the procedure impute Although it's name suggests that it is for categorical variables, you can set the scale to Numeric to handle the continuous case.

• this is SAS PROC CATREG, I'm guessing? – Ben Bolker Jun 21 '14 at 22:05
• @JKP: I had come across this command, actually. However, considering that my outome variable is binary, I am assuming logisic regression would be more appropriate than categorical regression (CATREG) - am I right? Also, the options in CATREG seem rather limited - you can only choose between excluding cases, replacing missing values with mean values or creating an extra category. – Puzzled Jun 23 '14 at 0:35

You may also consider the simple approach presented in the following paper:

Loh, P. L., & Wainwright, M. J. (2011). High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity. In Advances in Neural Information Processing Systems (pp. 2726-2734).