# Question about point mass prior and continuous distribution

Suppose we have a point mass prior,

$$\theta \sim \begin{cases} I(\theta=1) ,& prob=\frac{1}{2} \\ Gamma(c,c), & prob=\frac{1}{2} \end{cases}$$

Then if we are asked

$$\lim_{c \to \infty} P(\theta=1)$$

Now here is the issue, since Gamma is a continuous distribution, it seems that in the case of gamma,we will never have $$\theta=1$$.

To me it thus seems that regardless of the value of c, the $$p(\theta=1)=\frac{1}{2}$$

However, we also have that since expected value of a $$gamma(a,b)=\frac{a}{b}$$ so that the expected value of the gamma is 1 when we have $$a=b=c$$

But, by Markov, for $$X \sim Gamma(c,c)$$

$$\lim_{c \to \infty} Pr(|X-\mu| \lt \epsilon) \to 1$$ for any $$\epsilon \gt 0$$

So is $$\lim_{c \to \infty}P(\theta=1) =1$$ , or is $$\lim_{c \to \infty}P(\theta=1)=\frac{1}{2}$$

As even though the markov inequality holds, it is a continous distirbution, so we will never have it exactly equal to 1.

Thanks all

## 2 Answers

Your specified prior distribution is a mixture of a continuous and discrete part. Let $$I \sim \text{Bern}(1/2)$$ be the indicator that the parameter is taken from the gamma distribution. Then using the law of total probability you have:

\begin{equation} \begin{aligned} \mathbb{P}(\theta = 1|c) &= \mathbb{P}(\theta = 1|c, I=0) \cdot \mathbb{P}(I=0) + \mathbb{P}(\theta = 1|c, I=1) \cdot \mathbb{P}(I=1) \\[6pt] &= \frac{1}{2} \cdot \mathbb{P}(\theta = 1|c, I=0) + \frac{1}{2} \cdot \mathbb{P}(\theta = 1|c, I=1) \\[6pt] &= \frac{1}{2} + \frac{1}{2} \cdot \mathbb{P}(\theta = 1| \theta \sim \text{Ga}(c,c)) \\[6pt] &= \frac{1}{2}. \\[6pt] \end{aligned} \end{equation}

(Note that the last step comes from recognising that the gamma is a continuous distribution, so the probability of a specific point is zero under this distribution.) This result holds for all values of $$c$$, so you are correct that it also holds in the limit:

$$\lim_{c \rightarrow \infty} \mathbb{P}(\theta = 1|c) = \lim_{c \rightarrow \infty} \frac{1}{2} = \frac{1}{2}.$$

Your later use of Chebychev's inequality shows that as $$c \rightarrow \infty$$ you get $$\theta \rightarrow 1$$ (convergence in probability), but this does not change the fact that $$\mathbb{P}(\theta = 1|c) = 1/2$$ for all $$c > 0$$. (To understand the reason for this more fully, have a read about the distinction between convergence in probability and almost-sure convergence. Presumably the example is selected precisely to illustrate this distinction.)

Disclaimer The above answer is completely fine if the question is about the prior probability that $$θ=1$$. While there is no indication whatsoever in the text of the question to the alternative, namely that the question is about the posterior probability that $$θ=1$$, a recent exchange with students about a very similar question in my book, The Bayesian Choice (Exercise 5.9), leads me to propose the extension to the posterior setting.

In this posterior case, when $$X\sim \mathcal{E}(\theta)$$ [for instance] the probability that $$θ=1$$ is \begin{align*} \mathbb{P}(\theta=1|x)&= \frac{e^{-x}}{e^{-x}+\int_0^\infty \theta e^{-\theta x} \pi_c(\theta)\text{d}\theta}\\ &=\frac{e^{-x}}{e^{-x}+ \int_0^\infty \theta e^{-\theta x} c^c \theta^{c-1}e^{-c \theta} \Gamma(c)^{-1}\text{d}\theta}\\ &=\frac{e^{-x}}{e^{-x}+c^c (c+x)^{-c-1} \Gamma(c+1)\Gamma(c)^{-1}}\\ &=\frac{e^{-x}}{e^{-x}+c^{c+1} (c+x)^{-c-1}}\\ &=\frac{e^{-x}}{e^{-x}+ (1+x/c)^{-c-1}}\\ \end{align*} which converges to $$\frac{e^{-x}}{e^{-x}+e^{-x}}=\frac{1}{2}$$ when $$c$$ goes to infinity.