Confidence intervals with penalized likelihood I am trying to perform parameter estimation using something like a maximum likelihood ratio method, however I need to add a penalty term to constrain nuisance parameters which describe certain systematic uncertainties in the measurement process. So I have been digging around in the literature to try and better understand what is know about penalized likelihood estimation, but I cannot find anything I can understand regarding the construction of confidence intervals for my parameters of interest.
I get that a penalty term in the likelihood is quite similar to a Bayesian prior, and indeed if I wanted to then I could easily analyse my situation in a Bayesian way and use credible intervals instead of confidence intervals, however for this instance I want my intervals to have correct frequentist coverage properties, at least asymptotically (as in the usual profile likelihood case where I could rely on Wilks' theorem).
So, are there theorems similar to Wilks' that work with penalized likelihoods? Or with particular kinds of penalty terms? In the literature it looks to me like people mostly do numerical studies of coverage rather than rely on theorems, does this mean that there are no known "general purpose" theorems for this?
Edit: Alternatively, I suppose that I would be happy to be able to treat the penalty term in a fully Bayesian way, but nevertheless, in the end, still construct frequentist confidence intervals based on whatever estimator resulted, say the maximum of the marginal likelihood (i.e. with the nuisance parameter marginalised out). Looking around I do find some literature discussing the frequency properties of Bayesian point estimates, so I guess that it is possible that somewhere in this literature exists the sort of construction that I want?
 A: I don't think it's possible to provide meaningful confidence intervals in that case in a natural way.
Call $\theta$ the parameter and $\hat\theta$ the penalized likelihood estimator. Confidence intervals at 95% say that $P(|\theta-\hat\theta|\leq d|\theta)\geq0.95$ for all $\theta$. If you reframe the Bayesian idea in a back and white fashion: the reason why you need penalization is that not all $\theta$ are expected to really be "possible": you focus on the ones that you consider to be "possible". 
Outside of the "possible" zone, $\hat\theta$ is a very poor estimate. If the real $\theta$ is far from the possible zone $\hat\theta$ can be far from the real $\theta$ with high probability. For such $\theta$, $P(|\theta-\hat\theta|\leq d|\theta)\geq0.95$ is only true if $d$ is big. Since $d$ is required to not depend on $\theta$, you would just use the worst case scenario $d$ yielding useless confidence intervals. Actually, I think $d=+\infty$ if you consider $\theta$ going to infinity.
A way to fall back on a frequentist analysis in this case is to define a "possible" region $\Theta$ inspired from the penalization. One possibility is a region containing 99% of the weight of the prior. With $L^2$ regularized MLE for example, if this region is a ball whose radius is exactly the norm of the penalized estimate, then the frequentist MLE raw estimate is the same as the penalized one and lies on the border (sphere). With this method, you can say: if $\theta\in \Theta$ then $P(|\theta-\hat\theta|\leq d|\theta)\geq0.95$ with a meaningful $d$. It is a confidence interval with condition $\theta\in \Theta$.
