Bayesian consistency in compact uncountable parameter space Let $p(y_i \mid \theta)$ be the likelihood we are using of a single data point, $p(\theta)$ be the prior, and $f(y_i)$ the true distribution of the data. Also, let $\theta_0$ be the parameter that minimizes the Kullback-Leibler divergence between $p(y_i \mid \theta)$ and $f(y_i)$, which is
$$
KL(\theta) = E_f\left[ \log\left(\frac{f(y)}{p(y \mid \theta)}\right)\right] = \int \log\left(\frac{f(y)}{p(y \mid \theta)}\right) f(y) \text{d}y.
$$
Let $A_{\epsilon} = \{ \theta \in \Theta : \rho(\theta, \theta_0 ) < \epsilon \}$ be the $\epsilon$-ball about $\theta_0$. Let $B$ be another set that does not contain $\theta_0$, but is not necessarily disjoint from $A_{\epsilon}$.
It is easy to show that 
$$
KL(\theta_0) - KL(\theta) = E_f\left[ \log\left(\frac{p(y \mid \theta)}{p(y \mid \theta_0)}\right)\right] < 0
$$
but is it true that 
$$
E_f\left[\log \left(\frac{ p(y \mid \theta \in B)  }{  p(y \mid \theta' \in A_{\epsilon})  } \right) \right] < 0.
$$
Hopefully yes, because it would help me prove consistency in a compact parameter space.
 A: To mimic the proof from Bayesian Data Analysis we take
$$\log\left(\frac{p(\theta \in B \mid y)}{p(\theta \in A_\epsilon \mid y)}\right) = \log\left(\frac{p(\theta \in B)}{p(\theta \in A_\epsilon)}\right) + \sum_{i = 1}^n \log\left(\frac{p(y_i \mid \theta \in B)}{p(y_i \mid \theta \in A_\epsilon)}\right)$$
and we want to show that
$$E_f\left[\log\left(\frac{p(y_i \mid \theta \in B)}{p(y_i \mid \theta \in A_\epsilon)}\right)\right] < 0.$$
In your question you used the equality
$$p(y_i \mid \theta \in B) = \int_B p(y \mid \theta)\textrm{ d}\theta,$$
and this is not true. We would need to include a term with the density of $\theta$ in the integral as well. The quantity on the right just gets larger as $B$ expands, which is not what should happen with this likelihood: as $B$ includes values of $\theta$ that are further and further away from $\theta_0$, the expected value of this likelihood should start to shrink. If the likelihood and the integral were equal then the counter-example that I have below would imply that the sampler is inconsistent: that $\theta$ will end in $B$ rather than $A_\epsilon$ as $n\rightarrow \infty$.
I've tried to show that this expectation is negative but I have not been able to do so. The proof of the theorem in Bayesian Data Analysis appears in some papers and is very involved - I can't say that I understand it fully. The first page of (Bunke and Milhaud, 1998) states that the theorem is proved in (Berk, 1966) and (Berk, 1970). I have looked through these and I believe that they do prove the theorem but I don't have the level of expertise to check it properly. I thought this was a fascinating question.

Counter-Example It is not true that
$$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] < 0$$
subject to the conditions that you gave and that $\theta$ is defined on a compact space (which is a condition of the theorem in Bayesian Data Analysis that you mentioned in the original version of the post).
Here is my counter-example: Let $Y \sim U(0, \theta)$ so $p(y \mid \theta) = 1/\theta$, defined on the interval $0 \leq y \leq \theta$. Let $Y \sim U(0, 0.9)$ be the true distribution, so that $f(y) = 1/0.9$ for $0 \leq y \leq 0.9$. Let $\epsilon = 0.001$ and let $A_\epsilon = (0.9 - \epsilon, 0.9 + \epsilon)$ and $B = [0.9 + \epsilon, 1]$.
We have that $$\int_B p(y \mid \theta) \textrm{ d}\theta = \int_{0.9 + \epsilon}^1 \frac{1}{\theta} \thinspace \mathbb{I}_{\{0\leq y \leq \theta\}} \textrm{ d}\theta.$$ $\{0 \leq y \leq \theta\}$ is always true for $y$ in the range of $y$, $[0, 0.9]$, so $$\int_B p(y \mid \theta) \textrm{ d}\theta = \int_{0.9 + \epsilon}^1 \frac{1}{\theta} \textrm{ d}\theta = -\log(0.9 + \epsilon).$$ We also have $$\int_A p(y \mid \theta) \textrm{ d}\theta = \int_{\max(y, 0.9 - \epsilon)}^{0.9 + \epsilon} \frac{1}{\theta} \textrm{ d}\theta = \log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)).$$
Therefore
$$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] = \int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y.$$
We have that $\max(y, 0.9 - \epsilon) \geq 0.9 - \epsilon$, and after some calculations, which involve noting that $-\log(0.9 + \epsilon) > 0$ since $\epsilon = 0.001$, we have
$$\int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(\max(y, 0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y \geq \int_0^{0.9}\frac{-\log(0.9 + \epsilon)}{\log(0.9 + \epsilon) - \log(0.9 - \epsilon)}\frac{1}{0.9}\textrm{ d}y.$$
Substituting in $\epsilon = 0.001$ we have
$$E_f\left[ \log\left( \frac{\int_B p(y \mid \theta) \textrm{ d}\theta}{\int_{A_\epsilon} p(y \mid \theta) \textrm{ d}\theta} \right)\right] \geq 3.84 > 0.$$ 
