Question about point mass prior and continuous distribution

Question

Suppose we have a point mass prior,

$$\theta \sim \begin{cases} I(\theta=1) ,& prob=\frac{1}{2} \\ Gamma(c,c), & prob=\frac{1}{2} \end{cases}$$

Then if we are asked

$\lim_{c \to \infty} P(\theta=1)$

Now here is the issue, since Gamma is a continuous distribution, it seems that in the case of gamma,we will never have $\theta=1$.

To me it thus seems that regardless of the value of c, the $p(\theta=1)=\frac{1}{2}$

However, we also have that since expected value of a $gamma(a,b)=\frac{a}{b}$ so that the expected value of the gamma is 1 when we have $a=b=c$

But, by Markov, for $X \sim Gamma(c,c)$

$\lim_{c \to \infty} Pr(|X-\mu| \lt \epsilon) \to 1$ for any $\epsilon \gt 0$

So is $\lim_{c \to \infty}P(\theta=1) =1$ , or is $\lim_{c \to \infty}P(\theta=1)=\frac{1}{2}$

As even though the markov inequality holds, it is a continous distirbution, so we will never have it exactly equal to 1.

Thanks all

Answer 1

Your specified prior distribution is a mixture of a continuous and discrete part. Let $I \sim \text{Bern}(1/2)$ be the indicator that the parameter is taken from the gamma distribution. Then using the law of total probability you have:

$$\begin{equation} \begin{aligned} \mathbb{P}(\theta = 1|c) &= \mathbb{P}(\theta = 1|c, I=0) \cdot \mathbb{P}(I=0) + \mathbb{P}(\theta = 1|c, I=1) \cdot \mathbb{P}(I=1) \\[6pt] &= \frac{1}{2} \cdot \mathbb{P}(\theta = 1|c, I=0) + \frac{1}{2} \cdot \mathbb{P}(\theta = 1|c, I=1) \\[6pt] &= \frac{1}{2} + \frac{1}{2} \cdot \mathbb{P}(\theta = 1| \theta \sim \text{Ga}(c,c)) \\[6pt] &= \frac{1}{2}. \\[6pt] \end{aligned} \end{equation}$$

(Note that the last step comes from recognising that the gamma is a continuous distribution, so the probability of a specific point is zero under this distribution.) This result holds for all values of $c$, so you are correct that it also holds in the limit:

$$\lim_{c \rightarrow \infty} \mathbb{P}(\theta = 1|c) = \lim_{c \rightarrow \infty} \frac{1}{2} = \frac{1}{2}.$$

Your later use of Chebychev's inequality shows that as $c \rightarrow \infty$ you get $\theta \rightarrow 1$ (convergence in probability), but this does not change the fact that $\mathbb{P}(\theta = 1|c) = 1/2$ for all $c > 0$. (To understand the reason for this more fully, have a read about the distinction between convergence in probability and almost-sure convergence. Presumably the example is selected precisely to illustrate this distinction.)

Answer 2

Disclaimer The above answer is completely fine if the question is about the prior probability that $θ=1$. While there is no indication whatsoever in the text of the question to the alternative, namely that the question is about the posterior probability that $θ=1$, a recent exchange with students about a very similar question in my book, The Bayesian Choice (Exercise 5.9), leads me to propose the extension to the posterior setting.

In this posterior case, when $X\sim \mathcal{E}(\theta)$ [for instance] the probability that $θ=1$ is \begin{align*} \mathbb{P}(\theta=1|x)&= \frac{e^{-x}}{e^{-x}+\int_0^\infty \theta e^{-\theta x} \pi_c(\theta)\text{d}\theta}\\ &=\frac{e^{-x}}{e^{-x}+ \int_0^\infty \theta e^{-\theta x} c^c \theta^{c-1}e^{-c \theta} \Gamma(c)^{-1}\text{d}\theta}\\ &=\frac{e^{-x}}{e^{-x}+c^c (c+x)^{-c-1} \Gamma(c+1)\Gamma(c)^{-1}}\\ &=\frac{e^{-x}}{e^{-x}+c^{c+1} (c+x)^{-c-1}}\\ &=\frac{e^{-x}}{e^{-x}+ (1+x/c)^{-c-1}}\\ \end{align*} which converges to $$\frac{e^{-x}}{e^{-x}+e^{-x}}=\frac{1}{2}$$ when $c$ goes to infinity.