This is a "fleshed out" example given in a book written by Larry Wasserman All of statistics on Page 216 (12.8 Strengths and Weaknesses of Bayesian Inference). I basically provide what Wasserman doesn't in his book 1) an explanation for what is actually happening, rather than a throw away line; 2) the frequentist answer to the question, which Wasserman conveniently does not give; and 3) a demonstration that the equivalent confidence calculated using the same information suffers from the same problem.
In this example, he states the following situation
- An observation, X, with a Sampling distribution: $(X|\theta)\sim N(\theta,1)$
- Prior distribution of $(\theta)\sim N(0,1)$ (he actually uses a general $\tau^2$ for the variance, but his diagram specialises to $\tau^2=1$)
He then goes to show that, using a Bayesian 95% credible interval in this set-up eventually has 0% frequentist coverage when the true value of $\theta$ becomes arbitrarily large. For instance, he provides a graph of the coverage (p218), and checking by eye, when the true value of $\theta$ is 3, the coverage is about 35%. He then goes on to say:
...What should we conclude from all this? The important thing is to understand that frequentist and Bayesian methods are answering dierent questions. To combine prior beliefs with data in a principled way, use bayesian inference. To construct procedures with guaranteed long run performance, such as condence intervals, use frequentist methods... (p217)
And then moves on without any disection or explanation of why the Bayesian method performed apparently so bad. Further, he does not give a answer from the frequentist approach, just a broad brush statement about "the long-run" - a classical political tactic (emphasise your strength + others weakness, but never compare like for like).
I will show how the problem as stated $\tau=1$ can be formulated in frequentist/orthodox terms, and then show that the result using confidence intervals gives precisely the same answer as the Bayesian one. Thus any defect in the Bayesian (real or perceived) is not corrected by using confidence intervals.
Okay, so here goes. The first question I ask is what state of knowledge is described by the prior $\theta\sim N(0,1)$? If one was "ignorant" about $\theta$, then the appropriate way to express this is $p(\theta)\propto 1$. Now suppose that we were ignorant, and we observed $Y\sim N(\theta,1)$, independently of $X$. What would our posterior for $\theta$ be?
$$p(\theta|Y)\propto p(\theta)p(Y|\theta)\propto exp\Big(-\frac{1}{2}(Y-\theta)^2\Big)$$
Thus $(\theta|Y)\sim N(Y,1)$. This means that the prior distribution given in Wassermans example, is equivalent to having observed an iid copy of $X$ equal to $0$. Frequentist methods cannot deal with a prior, but it can be thought of as having made 2 observations from the sampling distribution, one equal to $0$, and one equal to $X$. Both problems are entirely equivalent, and we can actually give the frequentist answer for the question.
Because we are dealing with a normal distribution with known variance, the mean is a sufficient statistic for constructing a confidence interval for $\theta$. The mean is equal to $\overline{x}=\frac{0+X}{2}=\frac{X}{2}$ and has a sampling distribution
$$(\overline{x}|\theta)\sim N(\theta,\frac{1}{2})$$
Thus an $(1-\alpha)\text{%}$ CI is given by:
$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$
But, using The results of example 12.8 for Wasserman, he shows that the posterior $(1-\alpha)\text{%}$ credible interval for $\theta$ is given by:
$$cX\pm \sqrt{c}Z_{\alpha/2}$$.
Where $c=\frac{\tau^{2}}{1+\tau^{2}}$. Thus, plugging in the value at $\tau^{2}=1$ gives $c=\frac{1}{2}$ and the credible interval becomes:
$$\frac{1}{2}X\pm Z_{\alpha/2}\frac{1}{\sqrt{2}}$$
Which are exactly the same as the confidence interval! So any defect in the coverage exhibited by the Bayesian method, is not corrected by using the frequentist confidence interval! [If the frequentist chooses to ignore the prior, then to be a fair comparison, the Bayesian should also ignore this prior, and use the ignorance prior $p(\theta)\propto 1$, and the two intervals will still be equal - both $X \pm Z_{\alpha/2})$].
So what the hell is going on here? The problem is basically one of non-robustness of the normal sampling distribution. because the problem is equivalent to having already observed a iid copy, $X=0$. If you have observed $0$, then this is extremely unlikely to have occurred if the true value is $\theta=4$ (probability that $X\leq 0$ when $\theta=4$ is 0.000032). This explains why the coverage is so bad for large "true values", because they effectively make the implicit observation contained in the prior an outlier. In fact you can show that this example is basically equivalent to showing that the arithmetic mean has an unbounded influence function.
Generalisation. Now some people may say "but you only considered $\tau=1$, which may be a special case". This is not true: any value of $\tau^2=\frac{1}{N}$ $(N=0,1,2,3,\dots)$ can be interpreted as observing $N$ iid copies of $X$ which were all equal to $0$, in addition to the $X$ of the question. The confidence interval will have the same "bad" coverage properties for large $\theta$. But this becomes increasingly unlikely if you keep observing values of $0$ (and no rational person would continue to worry about large $\theta$ when you keep seeing $0$).