12
$\begingroup$

Let $f$, $g$ and $h$ be densities and suppose you have $x_i \sim h$, $i \in \mathbb{N}$. What happens to the likelihood ratio $$ \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} $$ as $n \rightarrow \infty$ ? (Does it converge? To what?)

For instance, we may assume $h = g$. The general case is also of interest.

$\endgroup$
3
  • $\begingroup$ Possible duplicate of Kullback-Leibler divergence WITHOUT information theory $\endgroup$
    – Xi'an
    Commented Jan 4, 2017 at 6:34
  • 4
    $\begingroup$ @Xi'an. I think adding this question to SE allows for the connection to be drawn across questions in the answer. While there may be answer similarities the questions are not he same. $\endgroup$
    – John
    Commented Jan 4, 2017 at 6:57
  • 1
    $\begingroup$ Thanks for the link. The question is not a duplicate, even though answers to my question may involve the Kullback-Leibler divergence. $\endgroup$
    – Olivier
    Commented Jan 4, 2017 at 14:57

2 Answers 2

17
+150
$\begingroup$

If one takes the logarithm of this product, $${\mathfrak{r}}=\log \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} = \sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$and turns it into an average $$\bar{\mathfrak{r}}_n=\frac{1}{n}\sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$the law of large numbers applies, hence one gets the almost sure convergence $$\bar{\mathfrak{r}}_n\stackrel{\text{a.s.}}{\longrightarrow}\mathbb{E}_h\left[\log \frac{f(X)}{g(X)}\right]=\int_\mathfrak{X} \log\frac{f(x)}{g(x)}\,h(x)\,\text{d}x$$assuming this integral is well-defined [counter-examples are easy to come by].

For instance, if $f$, $g$, and $h$ are densities for the Normal distributions with means $\mu_1$, $\mu_2$, and zero, respectively, all with variance one, the value of $$\int_\mathfrak{X} \log\frac{f(x)}{g(x)}\,h(x)\,\text{d}x$$ is $$\int_\mathfrak{X} \{(x-\mu_1)^2-(x-\mu^2_2)\}\,\varphi(x)\,\text{d}x= \mu_1^2-\mu^2_2\,.$$

Note also that, without the averaging, the product $$\prod_{i=1}^n \frac{f(x_i)}{h(x_i)}$$almost surely converges to zero (when $x_i\sim h(x)\,$). While the product $$\prod_{i=1}^n \frac{f(x_i)}{g(x_i)}$$almost surely converges to zero or infinity depending on whether $g$ or $f$ is closer to $h$ in the Kullback-Leibler divergence sense (when $x_i\sim h(x)\,$).

$\endgroup$
4
  • $\begingroup$ Could you conclude an answer? Is the last integral non-zero (say when $g=h$)? $\endgroup$
    – Olivier
    Commented Jan 4, 2017 at 15:01
  • 1
    $\begingroup$ Why should it be zero? If $f=g$ it is zero; if $f=h$ and $g\ne h$ it is positive. And if $g=h$ and $f\ne h$ it is negative. It can also be zero for $f\ne h$, $f\ne g$, and $g \ne h$ if $f$ and $g$ are at an equal distance from $h$. $\endgroup$
    – Xi'an
    Commented Jan 4, 2017 at 15:05
  • $\begingroup$ What do you mean at equal distance from $h$? Could you elaborate? Your answer is interesting but it doesnt't (yet) directly answer the question. $\endgroup$
    – Olivier
    Commented Jan 4, 2017 at 15:09
  • 1
    $\begingroup$ The main question. Because $\mathfrak{r}= n \mathfrak{r}_n$, it is the sign of the last integral that determines the asymptotic behaviour of the ratio. $\endgroup$
    – Olivier
    Commented Jan 4, 2017 at 15:17
0
$\begingroup$

Let $Z_n = \prod^n_i \frac{p(x)}{q(x)}$. Consider the quantity $$W_n = \frac{1}{n}log(Z_n) = \frac{1}{n}\sum_i^n log(\frac{p(x)}{q(x)})$$ By Strong Law of Large Numbers, $$\lim_{n \rightarrow \infty} W_n = E_{q(x)}[log(\frac{p(x)}{q(x)})] = \int_\mathcal{X} log(\frac{p(x)}{q(x)})q(x)dx$$

Since $log(a) < a-1 \ \forall a > 0 $ $ a \neq 1$ and that $\frac{p(x)}{q(x)} > 0$, $p(x) \neq q(x)$

$$W_n \rightarrow \int_\mathcal{X} log(\frac{p(x)}{q(x)})q(x)dx < \int_\mathcal{X} (\frac{p(x)}{q(x)} - 1)q(x)dx = \int_\mathcal{X} p(x)dx - \int_\mathcal{X} q(x)dx = 1 - 1 = 0$$ This gives us $$\lim_{n \rightarrow \infty} W_n < 0 \implies \lim_{n \rightarrow \infty} \frac{1}{n}log(Z_n) < 0 \implies \lim_{n \rightarrow \infty} n \cdot \frac{1}{n}log(Z_n) = -\infty \\ \implies \lim_{n \rightarrow \infty} log(Z_n) = -\infty \\ \implies \lim_{n \rightarrow \infty} Z_n = 0 \ \blacksquare$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.