Eliminating a nuisance parameter in likelihood ratio test

I am having an argument with a co-author about how to eliminate a nuisance parameter in a simple likelihood ratio test and am hoping that the community helps us settle it.

Our data $\mathbf{x}$ can be described by the likelihood functions $l(\mathbf{x};\theta_0,\eta)$ and $l(\mathbf{x};\theta_1,\eta)$ under hypotheses $H_0$ and $H_1$, where $\theta$ is the parameter of interest and $\eta$ is the random "nuisance" effect. We know the distribution function $f_\eta(\eta)$ that describes $\eta$.

The likelihood ratio is, of course, $$\Lambda(\mathbf{x},\eta)=\frac{l(\mathbf{x};\theta_0,\eta)}{l(\mathbf{x};\theta_1,\eta)}$$

Since we have $f_\eta(\eta)$, we can "average" out $\eta$ to obtain a simple test between two point hypotheses. We can then use Neyman-Pearson and calculate the value of threshold. Our argument stems from how to average it out. Since $\Lambda(\mathbf{x},\eta)$ is effectively a random variable that is a function of $\eta$, I think the following is the correct test statistic is

$$E_\eta\left[\frac{l(\mathbf{x};\theta_0,\eta)}{l(\mathbf{x};\theta_1,\eta)}\right]$$

while my co-author insists that the correct test statistic is

$$\frac{E_\eta[l(\mathbf{x};\theta_0,\eta)]}{E_\eta[l(\mathbf{x};\theta_1,\eta)]}$$ Who is right, and why?

• When you say "we know the distribution function that describes the nuisance parameter", I'm a little confused. In the LRT, parameters are fixed but unknown. If you're describing a parameter by a distribution it sounds like you're taking a Bayesian approach instead, which would lead people to wonder, why discuss LRT at all? Perhaps more details might help to clarify the situation. – Glen_b Apr 23 '15 at 4:54
• $\theta\in\{\theta_0,\theta_1\}$ is fixed but unknown, $\eta$ is random, and "uninteresting." Looks like van Trees has it the my co-author's way, Eq. (296) in Sec 2.5 "Composite Hypotheses"... – M.B.M. Apr 23 '15 at 5:27
• @Glen_b: I second your point that the only cases where this makes sense are the Bayesian and random effect situations. But this does not cover all settings with nuisance parameters. – Xi'an Apr 23 '15 at 6:20
• If $\eta\sim f_\eta(\eta)$, the "true" likelihood is $$\mathbb{E}[l(\mathbf{x};\theta,\eta)]=\int l(\mathbf{x};\theta,\eta) f_\eta(\eta)\,\text{d}\eta$$ so I would second your co-author's perspective. – Xi'an Apr 23 '15 at 6:23
• M.B.M. Are you talking about $\eta$ being a random effect? I'd probably have denoted those as $\eta_i$ or something along those lines -- but in any case you should be explicit about what you mean (in your question). – Glen_b Apr 23 '15 at 8:49

You have to compare two likelihoods, not doing the mean of a ratio. So your co-worker is right. Since you said that you have the distribution of $\eta$ it seems to me you are using a Bayesian approach.
$$\Lambda(\mathbf{x})=\frac{\sup_{\eta}L(\mathbf{x}|\theta_0,\eta)}{\sup_{\eta}L(\mathbf{x}|\theta_1,\eta)}$$