3
$\begingroup$

How do you calculate an estimate for the variance covariance matrix of a logistic regression with elastic net regularization?

Starting from the variance-covariance matrix of a plain vanilla logistic regression, how does the formula need to be augmented: $\hat{\Sigma}=-\left[\hat{p}(1-\hat{p})XX^{T}\right]^{-1}$
where $\hat{p}=\frac{1}{1+exp(X^{T}\hat{\Theta})}$
If you don't know the answer for elastic net, how would this be implemented for a logistic regression with ridge regularization?

Update: This link suggests OLS ridge is:
M = $(X'X+λI)^{−1}X'$
var(β)= ${\sigma}^{2}MM'$

Improve this question
$\endgroup$
7
  • $\begingroup$ I always thought there wasn't an analytical solution the covariance matrix in regularized regression and that standard errors need to be bootstrapped. No regularized regression routine I know of produces standard errors. I'd very much like to be wrong. $\endgroup$
    – shadowtalker
    Jul 9, 2014 at 3:54
  • $\begingroup$ Maybe, but I thought I'd read somewhere that there is a solution. A cursory search reveals this: stats.stackexchange.com/questions/106254/… $\endgroup$
    – Luke
    Jul 9, 2014 at 5:12
  • $\begingroup$ I'm tempted to put a bounty on that question for a real answer. Or maybe Community Wiki would be more appropriate? I'm not sure. $\endgroup$
    – shadowtalker
    Jul 9, 2014 at 12:03
  • $\begingroup$ There's also this: stackoverflow.com/questions/23660120/… $\endgroup$
    – Luke
    Jul 9, 2014 at 15:36
  • 1
    $\begingroup$ In general there's no closed-form expression for the coefficients in the logistic regression model, but this paper provides an equation for the variance-covariance matrix associated with a particular beta value in equation (16). This could get you started on deriving the updated variance-covariance matrix for a particular regularization term. $\endgroup$
    – josliber
    Jul 16, 2014 at 1:59

1 Answer 1

Reset to default
4
+50
$\begingroup$

As stated in this paper, Eq. (12), the $(i, j)$ element of the variance-covariance matrix for logistic regression is $\frac{\partial^2l(\hat\beta)}{\partial\hat\beta_i\partial\hat\beta_j}$, where $l(\hat\beta)$ is the log-conditional likelihood of coefficients $\hat\beta$ given the observed data.

As stated in this paper, Eq. 2.1, the log-conditional likelihood for ridge-regularized logistic regression with parameter $\lambda$ is defined as $l^\lambda(\hat\beta) := l(\hat\beta) - \lambda\hat\beta'\hat\beta$.

Hence, if we define $V(\hat\beta)$ to be the variance-covariance matrix of non-regularized logistic regression, then the variance-covariance matrix of ridge-regularized logistic regression is $V^\lambda(\hat\beta) = V(\hat\beta) - 2\lambda I$.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Bravo! Worth pointing out that the second paper states that "Unfortunately, this approximation to the variance... cannot be used directly to construct approximate confidence intervals around beta, since we have to take into account the bias of the estimate. Jackknife and bootstrapping might be possible methods to obtain more insight into the variability of beta". Kind of defeats the whole point for me. $\endgroup$
    – Luke
    Jul 16, 2014 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.