|
Let $p_1$ and $p_2$ be two distinct probability distributions. Define $$ L(q)=D(q||p_1)-D(q||p_2) $$ where $D$ is the usual Kullback-Leibler divergence. Assume the support of $p_2$ is included in the support of $p_1$. Is it true that $L(q)$ achieves a (global) maximum at $q=p_2$? Where can I find a proof of this fact (if true)? EDIT: I assume (a) discrete PMF (b) I look for the maximum in the subset $$ \Delta = \{ q| supp(q)\subseteq supp(p_1)\}. $$ Many thanks, |
||||
|
|
|
If we take the definition $$ D(q||p) = \int \log \left\{\frac{q(x)}{p(x)}\right\} q(x) \text{d}x\,, $$ your criterion is $$ L(q) = \int \log \left\{\frac{p_2(x)}{p_1(x)}\right\} q(x) \text{d}x $$ and looking at the normal example when $p_1$ is $\mathcal{N}(0,1)$ and $p_2$ is $\mathcal{N}(2,1)$ shows that $$ L(q) = \int \frac{1}{2}\left\{x^2-(x-2)^2\right\} q(x) \text{d}x = \int (x-2) q(x) \text{d}x\,, $$ which is not maximised for $q=p_2$. (There is no maximum.) Same thing if you consider two Poisson distributions, $\mathcal{P}(\lambda_1)$ and $\mathcal{P}(\lambda_2)$: $$ D(q) = \sum_{i=0}^\infty i \log(\lambda_2/\lambda_1) q_i = \log(\lambda_2/\lambda_1) \sum_{i=0}^\infty i q_i $$ (plus a constant) is not bounded in $q$ when $\lambda_2>\lambda_1$. Same thing with a comparison of binomials $\mathcal{B}(n,p_1)$ and $\mathcal{B}(n,p_2)$: $$ D(q) = \sum_{i=0}^n \log \frac{p_2^i(1-p_2)^{n-i}}{p_1^i(1-p_1)^{n-i}} q_i = \text{cst} + \log \frac{p_2(1-p_1)}{p_1(1-p_2)} \sum_{i=0}^n i q_i $$ which is maximised by the point mass at 0 or at $n$, depending on the sign of $\log \frac{p_2(1-p_1)}{p_1(1-p_2)}$. So your intuition seems to be wrong. |
||||
