Suppose I'm trying to estimate a large number of parameters from some high-dimensional data, using some kind of regularized estimates. The regularizer introduces some bias into the estimates, but it can still be a good trade-off because the reduction in variance should more than make up for it.
The problem comes when I want to estimate confidence intervals (e.g. using Laplace approximation or bootstrapping). Specifically, the bias in my estimates leads to bad coverage in my confidence intervals, which makes it hard to determine the frequentist properties of my estimator.
I've found some papers discussing this problem (e.g. "Asymptotic confidence intervals in ridge regression based on the Edgeworth expansion"), but the math is mostly above my head. In the linked paper, Equations 92-93 seem to provide a correction factor for estimates that were regularized by ridge regression, but I was wondering if there were good procedures that would work with a range of different regularizers.
Even a first-order correction would be extremely helpful.