# Is there a radical difference in how bayesian and frequentist approaches treat nuisance parameters?

The wiki article on credible intervals has the following statement:

credible intervals and confidence intervals treat nuisance parameters in radically different ways.

Credible intervals are based on the posterior distribution of the parameter and confidence interval is based on the maximum likelihood associated with the data generating process. It seems to me that how credible and confidence intervals are computed is not dependent on whether the parameters are nuisance or not. So, I am a bit puzzled by this statement.

PS: I am aware of alternative approaches to dealing with nuisance parameters under frequentist inference but I think they are less common than standard maximum likelihood. (See this question on the difference between partial, profile and marginal likelihoods.)

• Not an answer (because I don't know) but I just saw this new book on the Springer web site: "A Comparison of the Bayesian and Frequentist Approaches to Estimation" springer.com/statistics/statistical+theory+and+methods/book/… that (one would assume) may have some answers. Jul 30 '10 at 17:20

The fundamental difference is that in maximum likelihood based methods we can't integrate the nuisance parameters out (because the likelihood function is not a PDF and doesn't obey probability laws).

In maximum likelihood methods, the ideal way to deal with nuisance parameters is through marginal/conditional likelihoods, but these are defined differently from the question you linked. (There is a notion of an integrated (marginal/conditional) likelihood function as in the linked question, but this is not strictly the marginal likelihood function.)

Say you have a parameter of interest, $\theta$, a nuisance parameter, $\lambda$. Suppose a transformation of your data $X$ to $(Y, Z)$ exists such that either $Y$ or $Y|Z$ depends only on $\theta$. If $Y$ depends on $\theta$, then the joint density can be written

$f(Y, Z; \theta, \lambda) = f_{Y}(Y; \theta) f_{Z|Y}(Z|Y; \theta, \lambda)$.

In the latter case, we have

$f(Y, Z; \theta, \lambda) = f_{Y|Z}(Y|Z; \theta) f_{Z}(Z; \theta, \lambda)$.

In either case, the factor depending on $\theta$ alone is of interest. In the former, it's the basis for the definition of the marginal likelihood and in the latter, the conditional likelihood. The important point here is to isolate a component that depends on $\theta$ alone.

If we can't find such a transformation, we look at other likelihood functions to eliminate the nuisance. We usually start with a profile likelihood. To eliminate bias in the MLE, we try to obtain approximations for marginal or conditional likelihoods, usually through a "modified profile likelihood" function (yet another likelihood function!).

There are many details, but the short story is that the likelihood methods treat nuisance parameters quite differently than Bayesian methods. In particular, the estimated likelihoods don't account for uncertainty in the nuisance. Bayesian methods do account for it through the specification of a prior.

There are arguments in favor of an integrated likelihood function and lead to something resembling the Bayesian framework. If you're interested, I can dig up some references.

• I understand what you wrote but I am not sure that answers my question. The method you describe and the methods in the linked question in my ps avoid estimating the nuisance parameter on the grounds that the nuisance parameter is not of interest. Such an approach is fine but then without estimating the nuisance parameter we cannot construct a confidence interval for this parameter. Thus, the sentence in the wiki makes sense only if we estimate the nuisance parameter using standard maximum likelihood.
– user28
Jul 30 '10 at 22:45
• So my question is: if we estimate the nuisance parameter using maximum likelihood how is its treatment any different than estimating any other parameter. By the way there seems to be a typo in your eqns as the second term f(z|-) should not depend on theta, right?
– user28
Jul 30 '10 at 22:47
• The equation looks right to me -- I hope I'm not having a blind moment staring at the screen here. In each case, we have one component that depends on theta and the other (nuisance) component may or may not. The crucial point is that we must find a transformation that isolates theta to some extent to obtain either a marginal or conditional. When we can't, we must work with profile and other estimated likelihoods. We could argue that ML is losing information, but that's another question entirely. Still, perhaps this sheds light on the differences?
– ars
Jul 30 '10 at 23:33
• I am not sure how far we should take this discussion as SE is not a good outlet for discussions. I agree with what you said but I am not sure you are addressing the issue I raised. If you use a nuisance free likelihood how can you construct a confidence interval for the nuisance parameter? In any case, I should probably stop here and I will let you have the last comment.
– user28
Jul 31 '10 at 4:27
• Sorry, I wrote we don't estimate the nuisance, let me try again with some examples that address it. Partial likelihood just discards information about the nuisance. Profile replaces it with the MLE at fixed theta, but this doesn't account for uncertainty in lambda. Contrast with the Bayesian and specification of a prior. I think I get what you're saying that it's just estimation like any other parameter. But the treatment matters because it's how you account for the uncertainty due to the nuisance which affects the intervals, whether through the posterior or the likelihood.
– ars
Jul 31 '10 at 4:54