Let $f$, $g$ and $h$ be densities and suppose you have $x_i \sim h$, $i \in \mathbb{N}$. What happens to the likelihood ratio $$ \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} $$ as $n \rightarrow \infty$ ? (Does it converge? To what?)

For instance, we may assume $h = g$. The general case is also of interest.

  • $\begingroup$ Possible duplicate of Kullback-Leibler divergence WITHOUT information theory $\endgroup$ – Xi'an Jan 4 '17 at 6:34
  • 4
    $\begingroup$ @Xi'an. I think adding this question to SE allows for the connection to be drawn across questions in the answer. While there may be answer similarities the questions are not he same. $\endgroup$ – John Jan 4 '17 at 6:57
  • 1
    $\begingroup$ Thanks for the link. The question is not a duplicate, even though answers to my question may involve the Kullback-Leibler divergence. $\endgroup$ – Olivier Jan 4 '17 at 14:57

If one takes the logarithm of this product, $${\mathfrak{r}}=\log \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} = \sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$and turns it into an average $$\bar{\mathfrak{r}}_n=\frac{1}{n}\sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$the law of large numbers applies, hence one gets the almost sure convergence $$\bar{\mathfrak{r}}_n\stackrel{\text{a.s.}}{\longrightarrow}\mathbb{E}_h\left[\log \frac{f(X)}{g(X)}\right]=\int_\mathfrak{X} \log\frac{f(x)}{g(x)}\,h(x)\,\text{d}x$$assuming this integral is well-defined [counter-examples are easy to come by].

For instance, if $f$, $g$, and $h$ are densities for the Normal distributions with means $\mu_1$, $\mu_2$, and zero, respectively, all with variance one, the value of $$\int_\mathfrak{X} \log\frac{f(x)}{g(x)}\,h(x)\,\text{d}x$$ is $$\int_\mathfrak{X} \{(x-\mu_1)^2-(x-\mu^2_2)\}\,\varphi(x)\,\text{d}x= \mu_1^2-\mu^2_2\,.$$

Note also that, without the averaging, the product $$\prod_{i=1}^n \frac{f(x_i)}{h(x_i)}$$almost surely converges to zero (when $x_i\sim h(x)\,$). While the product $$\prod_{i=1}^n \frac{f(x_i)}{g(x_i)}$$almost surely converges to zero or infinity depending on whether $g$ or $f$ is closer to $h$ in the Kullback-Leibler divergence sense (when $x_i\sim h(x)\,$).

  • $\begingroup$ Could you conclude an answer? Is the last integral non-zero (say when $g=h$)? $\endgroup$ – Olivier Jan 4 '17 at 15:01
  • 1
    $\begingroup$ Why should it be zero? If $f=g$ it is zero; if $f=h$ and $g\ne h$ it is positive. And if $g=h$ and $f\ne h$ it is negative. It can also be zero for $f\ne h$, $f\ne g$, and $g \ne h$ if $f$ and $g$ are at an equal distance from $h$. $\endgroup$ – Xi'an Jan 4 '17 at 15:05
  • $\begingroup$ What do you mean at equal distance from $h$? Could you elaborate? Your answer is interesting but it doesnt't (yet) directly answer the question. $\endgroup$ – Olivier Jan 4 '17 at 15:09
  • 1
    $\begingroup$ The main question. Because $\mathfrak{r}= n \mathfrak{r}_n$, it is the sign of the last integral that determines the asymptotic behaviour of the ratio. $\endgroup$ – Olivier Jan 4 '17 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.