# LASSO to identify important variables in ordered logistic regression?

I have spent two days grappling with this question, and the range of ambiguous answers online has driven me to ask. I am working with R.

I have a dataset where my dependent variable is an ordered categorical variable with 5 levels ("dislike very much" to "like very much"). I intend to use the ordinal package for the actual regression analysis, but have been trying to decide on the best method to specify the model. From my reading it appears that stepwise methods are not a good option, and that a LASSO regression technique is a better method of selecting important variables. Is it acceptable to use a LASSO method to choose the variables to include and then to use these variables in a separate proportional odds regression? My primary interest is in the significance of the terms, rather than the size of the coefficient. I want to know which variables have a significant effect, in which direction, and in what order of importance. For this reason I would rather do the final modelling using a proportional odds glm than with a LASSO model.

• Your question makes it sound like you want to do two analyses. It is not clear what the first one (using the LASSO somehow) would be, however, you cannot select variables (even with the LASSO) w/ one analysis & this fit the final model using the selected variables on the same dataset. You need the shrinkage from the LASSO as part of the final model. – gung Feb 2 '14 at 18:06
• Yes, plus the variables selected by LASSO on a continuous variable might not be the right ones for an ordinal model. I am not sure if anyone has implemented LASSO for an ordinal model. That said, I think you can run a LASSO for variable selection and then use those for ordinal logistic, as long as you are honest about what you have done. I haven't seen any simulations of this; I wonder how much bias there would be in results. – Peter Flom Feb 2 '14 at 18:58
• Thanks so much for both of your comments. @gung yes I was worried that might be the case, that the shrinkage may change which variables are important. Peter Flom I might give it a try then, but what would your suggestion be otherwise? I am hesitant to use stepwise selection (your writings have helped to influence me there) but I am finding it difficult to find a reasonable alternative within my statistical ability.. – Jeremy Feb 2 '14 at 20:12
• The lasso estimates are equivalent to the posterior mode Bayesian estimates with double exponential $\sim \exp( - \lambda|\beta|)$ priors. The $\lambda$ scale parameter is related to the upper bound on the sum of the absolute values of the estimates in the lasso. This equivalency may be easier to utilize than trying to run maximum likelihood estimates with VERY WEIRD nonlinear constraints. – StasK Feb 2 '14 at 21:27
• @Jeremy My first choice would be to use substantive knowledge to select several reasonable models. If you really have no substantive knowledge, then I do know that LASSO has been developed for dichotomous logistic models, so you can model the ordinal outcomes as a set of dichotomous outcomes. – Peter Flom Feb 4 '14 at 13:30

## 2 Answers

Lasso is squared loss with l1-penalty, while ordinal logistic is the loss function, to which you can add the penalty of your choice. It seems you would like to have a model with an ordinal logistic loss AND an l1-penalty. This would be a legitimate model, although it is possible you might have to code it yourself since I don't know of any public implementations of this.

• Thanks Fabian. My main reason for wanting to use the Lasso was for model selection, and then actually run another ordinal logistic model with the variables that the Lasso determined to be important. But it seems like this two-step process may have theoretical flaws. Which leaves me with my original quest for the best method for model/variable selection for an ordinal logistic model, any suggestions? – Jeremy Feb 3 '14 at 8:08
• In that case the best option in my opinion would be to use l1-penalized ordinal logistic regression for feature selection and then (standard) ordinal logistic to learn the model. This procedure that learns an l1-penalized model for feature selection and then the same model without l1-regularization is called "debias" in the case of LASSO regression – Fabian Pedregosa Feb 4 '14 at 9:18
• @FabianPedregosa I've been warned explicitly against doing that, and that's also the impression I'm getting from the comments on the main question. Can you elaborate on the "debias" procedure? – shadowtalker Jul 19 '14 at 12:59
• It will be difficult to de-bias the severely biased estimates that result from trusting results learned from a properly penalized procedure and taking them to an unpenalized setting. It is best to use a single procedure, or to use pre-conditioning (model approximation) to represent the penalized model more simply. If you do that, then quadratic penalization probably works better than L1. – Frank Harrell Jul 19 '14 at 13:34
• P.S. Here are some references: citeulike.org/user/harrelfe/tag/model-approximation – Frank Harrell Jul 19 '14 at 14:12

You can try my package accSDA. I have not pushed the most recent changes to CRAN, but there is a function called ordASDA which implements LASSO based ordinal discriminant analysis (or ordinal regression).

The Readme on github has example code on how to run this, simple case would be:

# Prepare training and test set
train <- c(1:40,51:90,101:140)
Xtrain <- iris[train,1:4]

# normalize is a function in the package
nX <- normalize(Xtrain)
Xtrain <- nX\$Xc
Ytrain <- iris[train,5]
Xtest <- iris[-train,1:4]
Xtest <- normalizetest(Xtest,nX)
Ytest <- iris[-train,5]

# Define parameters for SDAD, i.e. ADMM optimization method
# Also try the SDAP and SDAAP methods, look at the documentation
# to read more about the parameters!
Om <- diag(4)+0.1*matrix(1,4,4) #elNet coef mat
gam <- 0.01
lam <- 0.01
method <- "SDAD"
q <- 2
control <- list(PGsteps = 100,
PGtol = c(1e-5,1e-5),
mu = 1,
maxits = 100,
tol = 1e-3,
ordinal = TRUE,
quiet = FALSE)

# Run the algorithm
res <- ASDA(Xt = Xtrain,
Yt = Ytrain,
Om = Om,
gam = gam ,
lam = lam,
q = q,
method = method,
control = control)


Just replace the ASDA with ordASDA or use the ordinal flag in the control list for ASDA. The iris dataset does not have an ordinal response, and the response has to be an integer in the set $$$1,n$$$ where n is the number of classes.