Bayesian consistency in compact uncountable parameter space

Let $$p(y_i \mid \theta)$$ be the likelihood we are using of a single data point, $$p(\theta)$$ be the prior, and $$f(y_i)$$ the true distribution of the data. Also, let $$\theta_0$$ be the parameter that minimizes the Kullback-Leibler divergence between $$p(y_i \mid \theta)$$ and $$f(y_i)$$, which is $$KL(\theta) = E_f\left[ \log\left(\frac{f(y)}{p(y \mid \theta)}\right)\right] = \int \log\left(\frac{f(y)}{p(y \mid \theta)}\right) f(y) \text{d}y.$$ Let $$A_{\epsilon} = \{ \theta \in \Theta : \rho(\theta, \theta_0 ) < \epsilon \}$$ be the $$\epsilon$$-ball about $$\theta_0$$. Let $$B$$ be another set that does not contain $$\theta_0$$, but is not necessarily disjoint from $$A_{\epsilon}$$.

It is easy to show that $$KL(\theta_0) - KL(\theta) = E_f\left[ \log\left(\frac{p(y \mid \theta)}{p(y \mid \theta_0)}\right)\right] < 0$$ but is it true that $$E_f\left[\log \left(\frac{ p(y \mid \theta \in B) }{ p(y \mid \theta' \in A_{\epsilon}) } \right) \right] < 0.$$ Hopefully yes, because it would help me prove consistency in a compact parameter space.

• Why is this a Bayesian question? You are only using the likelihood of the data given the parameter $\theta$, and never use the prior or posterior distribution of $\theta$. This seems an interesting question about misspecification of the likelihood - in the case where $f(y)$ only depends on $y$ and the true value of $\theta$ then $KL(\theta_0) = 0$. – Alex Dec 29 '18 at 11:09
• @Alex I edited away a lot of the details. They’re still visible if you check the logs, though. – Taylor Dec 29 '18 at 16:54

To mimic the proof from Bayesian Data Analysis we take $$\log\left(\frac{p(\theta \in B \mid y)}{p(\theta \in A_\epsilon \mid y)}\right) = \log\left(\frac{p(\theta \in B)}{p(\theta \in A_\epsilon)}\right) + \sum_{i = 1}^n \log\left(\frac{p(y_i \mid \theta \in B)}{p(y_i \mid \theta \in A_\epsilon)}\right)$$ and we want to show that $$E_f\left[\log\left(\frac{p(y_i \mid \theta \in B)}{p(y_i \mid \theta \in A_\epsilon)}\right)\right] < 0.$$ In your question you used the equality $$p(y_i \mid \theta \in B) = \int_B p(y \mid \theta)\textrm{ d}\theta,$$ and this is not true. We would need to include a term with the density of $$\theta$$ in the integral as well. The quantity on the right just gets larger as $$B$$ expands, which is not what should happen with this likelihood: as $$B$$ includes values of $$\theta$$ that are further and further away from $$\theta_0$$, the expected value of this likelihood should start to shrink. If the likelihood and the integral were equal then the counter-example that I have below would imply that the sampler is inconsistent: that $$\theta$$ will end in $$B$$ rather than $$A_\epsilon$$ as $$n\rightarrow \infty$$.

I've tried to show that this expectation is negative but I have not been able to do so. The proof of the theorem in Bayesian Data Analysis appears in some papers and is very involved - I can't say that I understand it fully. The first page of (Bunke and Milhaud, 1998) states that the theorem is proved in (Berk, 1966) and (Berk, 1970). I have looked through these and I believe that they do prove the theorem but I don't have the level of expertise to check it properly. I thought this was a fascinating question.

Counter-Example It is not true that $$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] < 0$$ subject to the conditions that you gave and that $$\theta$$ is defined on a compact space (which is a condition of the theorem in Bayesian Data Analysis that you mentioned in the original version of the post).

Here is my counter-example: Let $$Y \sim U(0, \theta)$$ so $$p(y \mid \theta) = 1/\theta$$, defined on the interval $$0 \leq y \leq \theta$$. Let $$Y \sim U(0, 0.9)$$ be the true distribution, so that $$f(y) = 1/0.9$$ for $$0 \leq y \leq 0.9$$. Let $$\epsilon = 0.001$$ and let $$A_\epsilon = (0.9 - \epsilon, 0.9 + \epsilon)$$ and $$B = [0.9 + \epsilon, 1]$$.

We have that $$\int_B p(y \mid \theta) \textrm{ d}\theta = \int_{0.9 + \epsilon}^1 \frac{1}{\theta} \thinspace \mathbb{I}_{\{0\leq y \leq \theta\}} \textrm{ d}\theta.$$ $$\{0 \leq y \leq \theta\}$$ is always true for $$y$$ in the range of $$y$$, $$[0, 0.9]$$, so $$\int_B p(y \mid \theta) \textrm{ d}\theta = \int_{0.9 + \epsilon}^1 \frac{1}{\theta} \textrm{ d}\theta = -\log(0.9 + \epsilon).$$ We also have $$\int_A p(y \mid \theta) \textrm{ d}\theta = \int_{\max(y, 0.9 - \epsilon)}^{0.9 + \epsilon} \frac{1}{\theta} \textrm{ d}\theta = \log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)).$$ Therefore $$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] = \int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y.$$ We have that $$\max(y, 0.9 - \epsilon) \geq 0.9 - \epsilon$$, and after some calculations, which involve noting that $$-\log(0.9 + \epsilon) > 0$$ since $$\epsilon = 0.001$$, we have $$\int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y \geq \int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y.$$ Substituting in $$\epsilon = 0.001$$ we have $$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] \geq 3.84 > 0.$$

• "and this is not true" ah of course (+1) – Taylor Dec 30 '18 at 15:52