# Confidence intervals with penalized likelihood

I am trying to perform parameter estimation using something like a maximum likelihood ratio method, however I need to add a penalty term to constrain nuisance parameters which describe certain systematic uncertainties in the measurement process. So I have been digging around in the literature to try and better understand what is know about penalized likelihood estimation, but I cannot find anything I can understand regarding the construction of confidence intervals for my parameters of interest.

I get that a penalty term in the likelihood is quite similar to a Bayesian prior, and indeed if I wanted to then I could easily analyse my situation in a Bayesian way and use credible intervals instead of confidence intervals, however for this instance I want my intervals to have correct frequentist coverage properties, at least asymptotically (as in the usual profile likelihood case where I could rely on Wilks' theorem).

So, are there theorems similar to Wilks' that work with penalized likelihoods? Or with particular kinds of penalty terms? In the literature it looks to me like people mostly do numerical studies of coverage rather than rely on theorems, does this mean that there are no known "general purpose" theorems for this?

Edit: Alternatively, I suppose that I would be happy to be able to treat the penalty term in a fully Bayesian way, but nevertheless, in the end, still construct frequentist confidence intervals based on whatever estimator resulted, say the maximum of the marginal likelihood (i.e. with the nuisance parameter marginalised out). Looking around I do find some literature discussing the frequency properties of Bayesian point estimates, so I guess that it is possible that somewhere in this literature exists the sort of construction that I want?

I don't think it's possible to provide meaningful confidence intervals in that case in a natural way.

Call $\theta$ the parameter and $\hat\theta$ the penalized likelihood estimator. Confidence intervals at 95% say that $P(|\theta-\hat\theta|\leq d|\theta)\geq0.95$ for all $\theta$. If you reframe the Bayesian idea in a back and white fashion: the reason why you need penalization is that not all $\theta$ are expected to really be "possible": you focus on the ones that you consider to be "possible".

Outside of the "possible" zone, $\hat\theta$ is a very poor estimate. If the real $\theta$ is far from the possible zone $\hat\theta$ can be far from the real $\theta$ with high probability. For such $\theta$, $P(|\theta-\hat\theta|\leq d|\theta)\geq0.95$ is only true if $d$ is big. Since $d$ is required to not depend on $\theta$, you would just use the worst case scenario $d$ yielding useless confidence intervals. Actually, I think $d=+\infty$ if you consider $\theta$ going to infinity.

A way to fall back on a frequentist analysis in this case is to define a "possible" region $\Theta$ inspired from the penalization. One possibility is a region containing 99% of the weight of the prior. With $L^2$ regularized MLE for example, if this region is a ball whose radius is exactly the norm of the penalized estimate, then the frequentist MLE raw estimate is the same as the penalized one and lies on the border (sphere). With this method, you can say: if $\theta\in \Theta$ then $P(|\theta-\hat\theta|\leq d|\theta)\geq0.95$ with a meaningful $d$. It is a confidence interval with condition $\theta\in \Theta$.

• Hmm, I see what you are saying, but my scenario is slightly different. The parameter for which I want an interval (say x) is not the same as the one I wish to penalise (say y). Usually I'd just construct confidence intervals for x from the profile likelihood, i.e. maximised over y, however here I don't have any constraint on y except for a Bayesian type penalty (i.e. there is no explicit control measurement or such a thing). So, sure, some values of y are improbable and I don't want the maximisation to consider them. I did think of what you suggest, i.e. just limiting the support of y... – Ben Farmer Nov 21 '17 at 11:56
• ...and then letting the maximisation just stop at the boundary, however since this is possible it makes me expect that it should also be possible with a less hard "boundary". I don't mind if the confidence interval coverage for x is ruined when the true y is far from where the prior says it is supposed to be, but if the true y is not too improbable under the prior I'd like to be able to get reasonable confidence intervals for x. – Ben Farmer Nov 21 '17 at 12:00
• Ok I see. $y$ could be seen as a hidden variable rather than a parameter. Then $x$ would be the only parameter and a full frequentist approach seems possible. – Benoit Sanchez Nov 21 '17 at 12:07
• Yeah. I could "fake" it and pretend that my prior represented an actual random process generating y, and indeed I have seen many people (particularly physicists) treat systematic uncertainties in this way, however I cannot shake the feeling that it is just not correct. Generally I'd say just do the Bayesian thing and then I can feel philosophically satisfied, however I can't help feeling that there must be some "correct" way to treat this situation in a fully frequentist way. – Ben Farmer Nov 21 '17 at 12:12
• Humm... yeah. Formally the true frequentist way deals with $P(...|x,y)$ while the hidden variable does $P(...|x)$. Beyond this I feel that if you allow $y$ to be anything, even the confidence interval of $x$ could be affected badly... – Benoit Sanchez Nov 21 '17 at 13:02