Penalized regression with zero-inflated models

Question

I'm currently building zero-inflated Poisson & negative binomial predictive models using the zeroinfl() function from the pscl package in R.

Incorporating penalized regressions into my model to account for shrinkage and variable selection is a priority. In addition I'd like to use penalization to avoid convergence issues due to perfect/quasi separation in my data (better than manually removing variables).

Question: Realizing that zero-inflated models $\neq$ hurdle models, for purposes of variable selection will my models be seriously biased if I first run separate run lasso (or elastic net) Poisson and logistic regressions with glmnet to select variables for the zeroinfl()?

Answer 1

I was looking for a ZIP model as well, and had trouble with mpath. I ended up coding my solutions in Matlab and it seems to work well, but I think it would be very easy in R (provided it is correct/ok). I may have overlooked something so let me know if you see any problems. My data has 80 obs and 550 features.

Recall that ZIP (and ZINB) can be solved via the EM algorithm. I am in the process of implementing this for ZINB, but my ZIP solution is as follows.

My understanding of EM -

Until Convergence: 
   step 1: calc expectations based on parameters
   step 2: maximize with respect to parameters.  

To do this in a Lasso framework, set your lambda sequences for Poisson model and the binomial model, call them I and J respectively

for i in I:
    for j in J:
    Until Convergence:
        step 1: calc expectations (estimate latent variable, call it zhat)
        step 2: estimate poisson regression with 1-zhat as weight.  
           estimate logistic regression with zhat as dv. 

The issue with step 2 - glmnet wants a factor or two columns to specify the proportion. Maybe just multiply the zhat by 100 and use that appropriately.

I picked i and j via grid search. Additionally, I do not go the full way down the sequence (maybe 30 on both). I also added in a break similar to the GLMNET package to stop before saturation.

Hope this helps.

Answer 2

As a coincidence, I just saw an update on the Cranberries package update feed about the mpath package.

From the package description (emphasis mine):

Algorithms for fitting model-based penalized coefficient paths. Currently the models include penalized Poisson, negative binomial, zero-inflated Poisson and zero-inflated negative binomial regression models. The penalties include least absolute shrinkage and selection operator (LASSO), smoothly clipped absolute deviation (SCAD) and minimax concave penalty (MCP), and each possibly combining with L_2 penalty.

It would appear to offer the kind of model you are looking for.

An alternative to consider is the mgcv package which ships with R. Whilst this is really for GAMs, it has family functions for the ZIP, plus one for ZIP where you can specify the linear predictor for the presence part and the count parts of the model separately. mgcv essentially fits penalized regression models; whilst you'd have to set up splines for the response variable you can turn on double shrinkage penalties in the bases for the spline such that entire terms can by shrunk out of the model in a similar way to the lasso (but these are, I believe fundamentally different approaches - I don't think the GAM approach has the geometric interpretation of why the lasso selects variables for example).

You can also add penalties to parametric terms if you don't want the splines, but I think you need to do more work to get that set up.