Variance-Covariance Matrix for $l_1$ regularized binomial logistic regression

Question

Given design matrix $X \in \mathbb{R}^{n \times p}$ and response vector $y \in \{ 0,1 \}^n$, I want to find the variance-covariance matrix of the coefficients $\hat{\beta}$ from an $l_1$-regularized logistic regression with regularization parameter $\lambda > 0$.

If I understand correctly, there is no closed-form solution for this as the penalized log-likelihood function $l(\beta \vert X, y, \lambda)$ is not differentiable. Approximations have been proposed in Tibshirani (1996), and Li and Fan (2001), but these only apply to the non-zero coefficients in $\hat{\beta}$. Bootstrap and Bayesian lasso (2010) are other methods to compute standard errors, but I think they would be too computationally intensive for my purposes. There is a question here with an answer that only applies to $l_2$-regularized logistic regression.

Are there any other fast and accurate alternatives for variance-covariance matrix computation of the coefficients in frequentist lasso logistic regression?

Answer 1

(This answer is more of a comment than a full answer, but I'm posting it here since I don't have enough rep to comment.)

This is a very hard question to give an good answer to. Even in the non-penalized case, the covariance estimate for the parameters is based on a normal approximation. When you start penalizing, you also enter the realm of "post-selection inference" which is an active area of research. The work on post-selection inference for GLMs (including logistic regression) is in its infancy, but see [1] for a recent reference on it. I believe the method described in this paper is implemented in the selectiveInference R package [2].

Even ignoring the mathematical difficulty, there are philosophical difficulties inherent in your question. Covariance matrices of estimators are tied to coverage under repeated sampling in the frequentist framework. If you had a new sample, you're not guaranteed to select the same variables, so can we even define "coverage" in a sensible way? There are many different (valid) ways to define coverage, each of which gives rise to a different school of thought about how post-selection inference should be defined and performed.

[1] J. Taylor, R. Tibshirani. "Post-selection inference for L1-penalized likelihood models." Canadian Journal of Statistics (to appear). http://doi.org/10.1002/cjs.11313

[2] https://cran.r-project.org/package=selectiveInference