# Variance Covariance for logit with elastic net

How do you calculate an estimate for the variance covariance matrix of a logistic regression with elastic net regularization?

Starting from the variance-covariance matrix of a plain vanilla logistic regression, how does the formula need to be augmented: $\hat{\Sigma}=-\left[\hat{p}(1-\hat{p})XX^{T}\right]^{-1}$
where $\hat{p}=\frac{1}{1+exp(X^{T}\hat{\Theta})}$
If you don't know the answer for elastic net, how would this be implemented for a logistic regression with ridge regularization?

Update: This link suggests OLS ridge is:
M = $(X'X+λI)^{−1}X'$
var(β)= ${\sigma}^{2}MM'$

• I always thought there wasn't an analytical solution the covariance matrix in regularized regression and that standard errors need to be bootstrapped. No regularized regression routine I know of produces standard errors. I'd very much like to be wrong. Jul 9, 2014 at 3:54
• Maybe, but I thought I'd read somewhere that there is a solution. A cursory search reveals this: stats.stackexchange.com/questions/106254/…
– Luke
Jul 9, 2014 at 5:12
• I'm tempted to put a bounty on that question for a real answer. Or maybe Community Wiki would be more appropriate? I'm not sure. Jul 9, 2014 at 12:03
• There's also this: stackoverflow.com/questions/23660120/…
– Luke
Jul 9, 2014 at 15:36
• In general there's no closed-form expression for the coefficients in the logistic regression model, but this paper provides an equation for the variance-covariance matrix associated with a particular beta value in equation (16). This could get you started on deriving the updated variance-covariance matrix for a particular regularization term. Jul 16, 2014 at 1:59

As stated in this paper, Eq. (12), the $(i, j)$ element of the variance-covariance matrix for logistic regression is $\frac{\partial^2l(\hat\beta)}{\partial\hat\beta_i\partial\hat\beta_j}$, where $l(\hat\beta)$ is the log-conditional likelihood of coefficients $\hat\beta$ given the observed data.
As stated in this paper, Eq. 2.1, the log-conditional likelihood for ridge-regularized logistic regression with parameter $\lambda$ is defined as $l^\lambda(\hat\beta) := l(\hat\beta) - \lambda\hat\beta'\hat\beta$.
Hence, if we define $V(\hat\beta)$ to be the variance-covariance matrix of non-regularized logistic regression, then the variance-covariance matrix of ridge-regularized logistic regression is $V^\lambda(\hat\beta) = V(\hat\beta) - 2\lambda I$.