2
$\begingroup$

How do you calculate an estimate for the variance covariance matrix of a logistic regression with elastic net regularization?

Starting from the variance-covariance matrix of a plain vanilla logistic regression, how does the formula need to be augmented: $\hat{\Sigma}=-\left[\hat{p}(1-\hat{p})XX^{T}\right]^{-1}$
where $\hat{p}=\frac{1}{1+exp(X^{T}\hat{\Theta})}$
If you don't know the answer for elastic net, how would this be implemented for a logistic regression with ridge regularization?

Update: This link suggests OLS ridge is:
M = $(X'X+λI)^{−1}X'$
var(β)= ${\sigma}^{2}MM'$

$\endgroup$
  • $\begingroup$ I always thought there wasn't an analytical solution the covariance matrix in regularized regression and that standard errors need to be bootstrapped. No regularized regression routine I know of produces standard errors. I'd very much like to be wrong. $\endgroup$ – shadowtalker Jul 9 '14 at 3:54
  • $\begingroup$ Maybe, but I thought I'd read somewhere that there is a solution. A cursory search reveals this: stats.stackexchange.com/questions/106254/… $\endgroup$ – Luke Jul 9 '14 at 5:12
  • $\begingroup$ I'm tempted to put a bounty on that question for a real answer. Or maybe Community Wiki would be more appropriate? I'm not sure. $\endgroup$ – shadowtalker Jul 9 '14 at 12:03
  • $\begingroup$ There's also this: stackoverflow.com/questions/23660120/… $\endgroup$ – Luke Jul 9 '14 at 15:36
  • 1
    $\begingroup$ In general there's no closed-form expression for the coefficients in the logistic regression model, but this paper provides an equation for the variance-covariance matrix associated with a particular beta value in equation (16). This could get you started on deriving the updated variance-covariance matrix for a particular regularization term. $\endgroup$ – josliber Jul 16 '14 at 1:59
4
+50
$\begingroup$

As stated in this paper, Eq. (12), the $(i, j)$ element of the variance-covariance matrix for logistic regression is $\frac{\partial^2l(\hat\beta)}{\partial\hat\beta_i\partial\hat\beta_j}$, where $l(\hat\beta)$ is the log-conditional likelihood of coefficients $\hat\beta$ given the observed data.

As stated in this paper, Eq. 2.1, the log-conditional likelihood for ridge-regularized logistic regression with parameter $\lambda$ is defined as $l^\lambda(\hat\beta) := l(\hat\beta) - \lambda\hat\beta'\hat\beta$.

Hence, if we define $V(\hat\beta)$ to be the variance-covariance matrix of non-regularized logistic regression, then the variance-covariance matrix of ridge-regularized logistic regression is $V^\lambda(\hat\beta) = V(\hat\beta) - 2\lambda I$.

$\endgroup$
  • $\begingroup$ Bravo! Worth pointing out that the second paper states that "Unfortunately, this approximation to the variance... cannot be used directly to construct approximate confidence intervals around beta, since we have to take into account the bias of the estimate. Jackknife and bootstrapping might be possible methods to obtain more insight into the variability of beta". Kind of defeats the whole point for me. $\endgroup$ – Luke Jul 16 '14 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.