Let $p(y_i \mid \theta)$ be the likelihood we are using of a single data point, $p(\theta)$ be the prior, and $f(y_i)$ the true distribution of the data. Also, let $\theta_0$ be the parameter that minimizes the Kullback-Leibler divergence between $p(y_i \mid \theta)$ and $f(y_i)$, which is $$ KL(\theta) = E_f\left[ \log\left(\frac{f(y)}{p(y \mid \theta)}\right)\right] = \int \log\left(\frac{f(y)}{p(y \mid \theta)}\right) f(y) \text{d}y. $$ Let $A_{\epsilon} = \{ \theta \in \Theta : \rho(\theta, \theta_0 ) < \epsilon \}$ be the $\epsilon$-ball about $\theta_0$. Let $B$ be another set that does not contain $\theta_0$, but is not necessarily disjoint from $A_{\epsilon}$.

It is easy to show that $$ KL(\theta_0) - KL(\theta) = E_f\left[ \log\left(\frac{p(y \mid \theta)}{p(y \mid \theta_0)}\right)\right] < 0 $$ but is it true that $$ E_f\left[\log \left(\frac{ p(y \mid \theta \in B) }{ p(y \mid \theta' \in A_{\epsilon}) } \right) \right] < 0. $$ Hopefully yes, because it would help me prove consistency in a compact parameter space.

  • $\begingroup$ Why is this a Bayesian question? You are only using the likelihood of the data given the parameter $\theta$, and never use the prior or posterior distribution of $\theta$. This seems an interesting question about misspecification of the likelihood - in the case where $f(y)$ only depends on $y$ and the true value of $\theta$ then $KL(\theta_0) = 0$. $\endgroup$ – Alex Dec 29 '18 at 11:09
  • $\begingroup$ @Alex I edited away a lot of the details. They’re still visible if you check the logs, though. $\endgroup$ – Taylor Dec 29 '18 at 16:54

To mimic the proof from Bayesian Data Analysis we take $$\log\left(\frac{p(\theta \in B \mid y)}{p(\theta \in A_\epsilon \mid y)}\right) = \log\left(\frac{p(\theta \in B)}{p(\theta \in A_\epsilon)}\right) + \sum_{i = 1}^n \log\left(\frac{p(y_i \mid \theta \in B)}{p(y_i \mid \theta \in A_\epsilon)}\right)$$ and we want to show that $$E_f\left[\log\left(\frac{p(y_i \mid \theta \in B)}{p(y_i \mid \theta \in A_\epsilon)}\right)\right] < 0.$$ In your question you used the equality $$p(y_i \mid \theta \in B) = \int_B p(y \mid \theta)\textrm{ d}\theta,$$ and this is not true. We would need to include a term with the density of $\theta$ in the integral as well. The quantity on the right just gets larger as $B$ expands, which is not what should happen with this likelihood: as $B$ includes values of $\theta$ that are further and further away from $\theta_0$, the expected value of this likelihood should start to shrink. If the likelihood and the integral were equal then the counter-example that I have below would imply that the sampler is inconsistent: that $\theta$ will end in $B$ rather than $A_\epsilon$ as $n\rightarrow \infty$.

I've tried to show that this expectation is negative but I have not been able to do so. The proof of the theorem in Bayesian Data Analysis appears in some papers and is very involved - I can't say that I understand it fully. The first page of (Bunke and Milhaud, 1998) states that the theorem is proved in (Berk, 1966) and (Berk, 1970). I have looked through these and I believe that they do prove the theorem but I don't have the level of expertise to check it properly. I thought this was a fascinating question.

Counter-Example It is not true that $$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] < 0$$ subject to the conditions that you gave and that $\theta$ is defined on a compact space (which is a condition of the theorem in Bayesian Data Analysis that you mentioned in the original version of the post).

Here is my counter-example: Let $Y \sim U(0, \theta)$ so $p(y \mid \theta) = 1/\theta$, defined on the interval $0 \leq y \leq \theta$. Let $Y \sim U(0, 0.9)$ be the true distribution, so that $f(y) = 1/0.9$ for $0 \leq y \leq 0.9$. Let $\epsilon = 0.001$ and let $A_\epsilon = (0.9 - \epsilon, 0.9 + \epsilon)$ and $B = [0.9 + \epsilon, 1]$.

We have that $$\int_B p(y \mid \theta) \textrm{ d}\theta = \int_{0.9 + \epsilon}^1 \frac{1}{\theta} \thinspace \mathbb{I}_{\{0\leq y \leq \theta\}} \textrm{ d}\theta.$$ $\{0 \leq y \leq \theta\}$ is always true for $y$ in the range of $y$, $[0, 0.9]$, so $$\int_B p(y \mid \theta) \textrm{ d}\theta = \int_{0.9 + \epsilon}^1 \frac{1}{\theta} \textrm{ d}\theta = -\log(0.9 + \epsilon).$$ We also have $$\int_A p(y \mid \theta) \textrm{ d}\theta = \int_{\max(y, 0.9 - \epsilon)}^{0.9 + \epsilon} \frac{1}{\theta} \textrm{ d}\theta = \log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)).$$ Therefore $$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] = \int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y.$$ We have that $\max(y, 0.9 - \epsilon) \geq 0.9 - \epsilon$, and after some calculations, which involve noting that $-\log(0.9 + \epsilon) > 0$ since $\epsilon = 0.001$, we have $$\int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y \geq \int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y.$$ Substituting in $\epsilon = 0.001$ we have $$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] \geq 3.84 > 0.$$

| cite | improve this answer | |
  • 2
    $\begingroup$ "and this is not true" ah of course (+1) $\endgroup$ – Taylor Dec 30 '18 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.