When using glmnet how to report p-value significance to claim significance of predictors?

Question

I have a large set of predictors (more than 43,000) for predicting a dependent variable which can take 2 values (0 or 1). The number of observations is more than 45,000. Most of the predictors are unigrams, bigrams and trigrams of words, so there is high degree of collinearity among them. There is a lot of sparsity in my dataset as well. I am using the logistic regression from the glmnet package, which works for the kind of dataset I have. My problem is how can I report p-value significance of the predictors. I do get the beta coefficient, but is there a way to claim that the beta coefficients are statistically significant?

Here is my code:

library('glmnet')
data <- read.csv('datafile.csv', header=T)
mat = as.matrix(data)
X = mat[,1:ncol(mat)-1] 
y = mat[,ncol(mat)]
fit <- cv.glmnet(X,y, family="binomial")

Another question is: I am using the default alpha=1, lasso penalty which causes the additional problem that if two predictors are collinear the lasso will pick one of them at random and assign zero beta weight to the other. I also tried with ridge penalty (alpha=0) which assigns similar coefficients to highly correlated variables rather than selecting one of them. However, the model with lasso penalty gives me a much lower deviance than the one with ridge penalty. Is there any other way that I can report both predictors which are highly collinear?

Answer 1

There is a new paper, A Significance Test for the Lasso, including the inventor of LASSO as an author that reports results on this problem. This is a relatively new area of research, so the references in the paper cover a lot of what is known at this point.

As for your second question, have you tried $\alpha \in (0,1)$? Often there is a value in this middle range that achieves a good compromise. This is called Elastic Net regularization. Since you are using cv.glmnet, you will probably want to cross-validate over a grid of $(\lambda, \alpha)$ values.

Answer 2

Post-selection inference is a very active topic of statistical research. In my view, an issue with the method described in A Significance Test for the Lasso is that stringent assumptions are required (reproduced from here):

1. The linear model is correct.
2. The variance is constant.
3. The errors have a Normal distribution.
4. The parameter vector is sparse.
5. The design matrix has very weak collinearity. This is usually stated in the form of incoherence, eigenvalue restrictions or incompatibility assumptions.

The approach I've found to be useful - as long as there is sufficient data available - is data splitting. The idea of data splitting goes back to at least Moran (1974) and simply entails dividing the data randomly into two sets, making modeling choices on the first set, and making inference on the second set.

So in this case you would split the data into two, do variable selection on the first half, then (assuming you have $n > p$) use standard regression techniques on the second half to determine the statistical significance of the coefficients. Of course, assumptions are still required at both stage but they may be easier to satisfy for each stage individually.

You mention that the covariates are uni-, bi-, and tri-grams so they are highly collinear. So in this case applying the Lasso in the first stage would also violate assumptions - in particular, #5 from above. So to make such an approach genuinely useful and theoretically sound you would need to do some sort of pre-Lasso collinearity screening.

Answer 3

Maybe have a look at CRAN package hdi, that one provides inference for high-dimensional models and should do the trick... The full methods are are a bit tedious to repeat here (there are several, and it's still quite an active area of research), but they are well described in this paper: http://projecteuclid.org/euclid.ss/1449670857 (If you publicly post some test data, or simulate some data, I can also give you a concrete example)