I am building an application which involves multinomial logistic regression models with the elastic net penalty using the glmnet-library on automatically collected data in R. My interest in particular relates to the predicted probabilities: As they are central to my application it would be useful to provide a confidence interval to quantify their uncertainty. However as far as I reviewed the literature this is quite difficult as standard errors for the elastic net are not straightforward to calculate or interprete due to the bias introduced by the shrinkage as explained for example here on page 18:

It is a very natural question to ask for standard errors of regression coefficients or other estimated quantities. In principle such standard errors can easily be calculated, e.g. using the bootstrap. Still, this package deliberately does not provide them. The reason for this is that standard errors are not very meaningful for strongly biased estimates such as arise from penalized estimation methods.

However compared to many other utilizations of the elastic net found in the literature my situation is not necessarily very high dimensional: The number of covariates is at maximum 100 and while it might be the case that the observational count is in extreme situations below that value the number of observations mostly ranges from 1 000 to 1 000 000, so that we can consider p to be fixed while n goes to infinity. The aforementioned paper briefly mentions that in such situations

[r]eliable confidence intervals around the penalized estimates can be obtained in the case of low dimensional models using the standard generalized linear model theory [...].

As far as I interpret this sentence it would just imply that I assume the bias to be negligible and compute standard errors as usual, from which I can estimate the confidence intervals for the predicted probabilities using the delta method.

Unfortunately I can not find any comprehensive discussion of the n > p case for the elastic net so I am not sure whether there is any real justification behind this argument (for example related to the asymptotics of the estimator). Based on this I would like to know if there is any way of calculating confidence intervals for the predicted probabilities and if or when it is appropriate to just assume that the bias is small enough to ignore it.

Edit: One more thing that just came to my mind is a kind of vulgar bayesian approach to at least establish that consistency exists for n going to infinity and fixed p. Zhou and Haste (2005) note that the regularization given by their Elastic Net is de facto equivalent to the MAP of a bayesian model with a prior distribution which is a compromise between the Gaussian and Laplacian priors on the beta-coefficients. The Bernstein–von Mises theorem on the other hand states that the posterior distribution for unknown quantities in non-high-dimensional settings (as mentioned in the last sentences of Diaconis and Freedman (1986)) is effectively asymptotically independent of the prior distribution and therefore coincides with the MLE, therefore asymptotically the standard errors and the classical methods of creating confidence intervals for predictive probabilities can be used. Is this correct? This still does not help with quantifying the size of the bias in finite samples of comparatively small size, but it would at least be a first step.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.