# How to select a subset of variables from my original long list in order to perform logistic regression analysis?

My situation:

• small sample size: 116
• binary outcome variable
• long list of explanatory variables: 44
• explanatory variables did not come from the top of my head; their choice was based on the literature.

Statistical test chosen: logistic regression

I need to find the variables that best explain variations in the outcome variable (I am not interested in making predictions).

The problem: This question is a follow-up on the 2 questions listed below. From them, I got that performing automated stepwise regression has its downsides. Anyway, it seems that my sample size would be too small for that. It seems that my sample is also too small to enter all variables at once (using the SPSS 'Enter' method). This leaves me with my issue unresolved: how can I select a subset of variables from my original long list in order to perform multivariate logistic regression analysis?

UPDATE1: I am not an statistician, so I would appreciate if jargons can be reduced to the minimum. I am working with SPSS and am not familiar with other packages, so options that could be run with that software would be highly preferable.

UPDATE2: It seems that SPSS does not support LASSO for logistic regression. So following one of your suggestions, I am now struggling with R. I have passed through the basics, and managed to run a univariate logistic regression routine successfully using the glm code. But as I tried glmnet with the same dataset, I am receiving an error message. How could I fix it? Below is the code I used, followed by the error message:

data1 <- read.table("C:\\\data1.csv",header=TRUE,sep=";",na.string=99:9999)

y <- data1[,1]

x <- data1[,2:45]

glmnet(x,y,family="binomial",alpha=1)

**in lognet(x, is.sparse, ix, jx, y, weights, offset, alpha, nobs,  :
(list) object cannot be coerced to type 'double'**


UPDATE3: I got another error message, now related to missing values. My question concerning that matter is here.

• What is the sample size for the smaller of the two outcome categories? – Matt Reichenbach Jun 11 '14 at 14:57
• @Matt Reichenback: The sample size for the smaller of the two outcome categories is 32. – Puzzled Jun 11 '14 at 21:56
• Hard to be sure without a reproducible example, but y needs to be a factor; so, if it's not already, y <- as.factor(dat1[ ,1]). Then x needs to be a matrix, so x <- as.matrix(dat1[ ,2:45]). – Scortchi - Reinstate Monica Jun 20 '14 at 11:52
• @Scortchi: Just tried that. But now I am getting the following error message: Error in lognet(x, is.sparse, ix, jx, y, weights, offset, alpha, nobs, : NA/NaN/Inf in foreign function call (arg 5). Maybe some problem with my missing values? What do you think? – Puzzled Jun 20 '14 at 12:32
• Remove cases or predictors with missing values to find out. As to the best way to deal with missing values in LASSO, it merits a question to itself IMO, if it hasn't already been asked. If it's just a few predictors of the 44 with missing values, I'd probably favour excluding those predictors rather than trying to impute the missing values. – Scortchi - Reinstate Monica Jun 20 '14 at 13:43

You can perform selection and logistic regression simultaneously using the LASSO or Elastic Net regression algorithms. The basic idea behind LASSO is to solve the $l_1$-penalized optimization problem $$\min_{\beta} \{ l(\beta) + \lambda||\beta||_1 \},$$ where $l(\cdot)$ is the likelihood function. Popular implementations, e.g. glmnet, efficiently solve for a grid of $\lambda$ values. This is useful because we usually don't know $\lambda$ a priori and need to apply some type of cross-validation. If you have correlated features then it helps to add some $l_2$ (ridge) penalty, which is the idea behind the Elastic Net.

Since you don't have a lot of data, I think this is probably your best bet. If you want to use a separate variable selection stage you will need to choose a metric (e.g. deviance of single-variable regression) and also a threshold. The LASSO gives you only one parameter to tune and operates within the context of multivariable logistic regression models directly.

EDIT: The question now specifically requests an approach that is implemented in SPSS. As I don't have/use that software I don't know whether lasso logistic regression is implemented. Perhaps someone can let us know in the comments.

• I seriously doubt that logistic regression lasso is implemented in SPSS, which is what the OP apparently has to use. – StasK Jun 12 '14 at 2:48
• SPSS does have LASSO and Elastic Net since version 17. Does it support logistic regression? I'm not sure. Regardless, the OP mentions SPSS, but I don't think it's apparent that SPSS must be used. Anyone can use R, it's free. I gave a link to an R package, but the question was about regression, not software. – MichaelJ Jun 12 '14 at 3:34
• @StasK, your statement RE: Excel is false. I don't have it on my computer and it costs $. – MichaelJ Jun 12 '14 at 14:40 • The specific request for SPSS was a later edit to the question that wasn't there when I gave my answer. Hence my later edit. – MichaelJ Jun 14 '14 at 12:59 • From what I could get about LASSO, it seems a good option. But I am struggling to experiment with it on SPSS. Among the types of regression it offers, there is CATREG (optimal scaling, categorical regression), where it gives the LASSO option for regularization. Is this what I am aiming at? If so, I will later ask for help with setting up and error messages that I am getting. The logistic regression routine does not offer LASSO as an option. Obs: SPSS' help section is far from being as straghtfoward as non-statistitians need, in my opinion. – Puzzled Jun 16 '14 at 13:45 Besides the excellent suggestions about using shrinkage approaches. Quadratic penalization should also be considered (we have a case study on this in J Clinical Epidemiology, first author Moons). Other than that, data reduction or redundancy analysis (all masked to$Y$) can play an important role, e.g., combining variables that are hard to separate. Variable clustering and principal components are two of many data reduction methods. With the number of events available, the 15:1 rule would indicate that reduction is needed down to two factors (masked to$Y\$).

• (+1) Just like to add that data reduction guided by subject-matter knowledge & common sense alone can also be effective - which might be appealing to a non-statistician. – Scortchi - Reinstate Monica Jun 12 '14 at 17:24
• Quadratic penalization won't select a subset of variables, no? – MichaelJ Jun 14 '14 at 13:04
• Quadratic penalization is usually called ridge regression, so I suggest searching using the latter keywords. As pointed out, quadratic penalization won't select a subset of variables, which was originally specified as a need. If predictive accuracy is the main concern then ridge regression (quadratic penalization) is probably the way to go. – MichaelJ Jun 16 '14 at 16:21
• See Moons et al, J Clin Epi 57:2363-70, 2004. – Frank Harrell Jun 16 '14 at 16:26
• @Puzzled: E.g. the response is likely related to a patient's fatness, & you use body-mass index as a predictor rather than height & weight. Or it's likely related to a increase in the amount of rainfall, & you form a predictor from the trend component of a time series model fitted to daily measurements. Or it's likely related to an individual's wealth, & you form a predictor by adding savings, investments & property value. Or it's likely related to a student's academic ability, & you form a predictor from a weighted average of test scores in different subjects. – Scortchi - Reinstate Monica Jun 20 '14 at 9:58