Suppose we have a point mass prior,

$$\theta \sim \begin{cases} I(\theta=1) ,& prob=\frac{1}{2} \\ Gamma(c,c), & prob=\frac{1}{2} \end{cases}$$

Then if we are asked

$\lim_{c \to \infty} P(\theta=1)$

Now here is the issue, since Gamma is a continuous distribution, it seems that in the case of gamma,we will never have $\theta=1$.

To me it thus seems that regardless of the value of c, the $p(\theta=1)=\frac{1}{2}$

However, we also have that since expected value of a $gamma(a,b)=\frac{a}{b}$ so that the expected value of the gamma is 1 when we have $a=b=c$

But, by Markov, for $X \sim Gamma(c,c)$

$\lim_{c \to \infty} Pr(|X-\mu| \lt \epsilon) \to 1$ for any $\epsilon \gt 0$

So is $\lim_{c \to \infty}P(\theta=1) =1$ , or is $\lim_{c \to \infty}P(\theta=1)=\frac{1}{2}$

As even though the markov inequality holds, it is a continous distirbution, so we will never have it exactly equal to 1.

Thanks all


Your specified prior distribution is a mixture of a continuous and discrete part. Let $I \sim \text{Bern}(1/2)$ be the indicator that the parameter is taken from the gamma distribution. Then using the law of total probability you have:

$$\begin{equation} \begin{aligned} \mathbb{P}(\theta = 1|c) &= \mathbb{P}(\theta = 1|c, I=0) \cdot \mathbb{P}(I=0) + \mathbb{P}(\theta = 1|c, I=1) \cdot \mathbb{P}(I=1) \\[6pt] &= \frac{1}{2} \cdot \mathbb{P}(\theta = 1|c, I=0) + \frac{1}{2} \cdot \mathbb{P}(\theta = 1|c, I=1) \\[6pt] &= \frac{1}{2} + \frac{1}{2} \cdot \mathbb{P}(\theta = 1| \theta \sim \text{Ga}(c,c)) \\[6pt] &= \frac{1}{2}. \\[6pt] \end{aligned} \end{equation}$$

(Note that the last step comes from recognising that the gamma is a continuous distribution, so the probability of a specific point is zero under this distribution.) This result holds for all values of $c$, so you are correct that it also holds in the limit:

$$\lim_{c \rightarrow \infty} \mathbb{P}(\theta = 1|c) = \lim_{c \rightarrow \infty} \frac{1}{2} = \frac{1}{2}.$$

Your later use of Chebychev's inequality shows that as $c \rightarrow \infty$ you get $\theta \rightarrow 1$ (convergence in probability), but this does not change the fact that $\mathbb{P}(\theta = 1|c) = 1/2$ for all $c > 0$. (To understand the reason for this more fully, have a read about the distinction between convergence in probability and almost-sure convergence. Presumably the example is selected precisely to illustrate this distinction.)


Disclaimer The above answer is completely fine if the question is about the prior probability that $θ=1$. While there is no indication whatsoever in the text of the question to the alternative, namely that the question is about the posterior probability that $θ=1$, a recent exchange with students about a very similar question in my book, The Bayesian Choice (Exercise 5.9), leads me to propose the extension to the posterior setting.

In this posterior case, when $X\sim \mathcal{E}(\theta)$ [for instance] the probability that $θ=1$ is \begin{align*} \mathbb{P}(\theta=1|x)&= \frac{e^{-x}}{e^{-x}+\int_0^\infty \theta e^{-\theta x} \pi_c(\theta)\text{d}\theta}\\ &=\frac{e^{-x}}{e^{-x}+ \int_0^\infty \theta e^{-\theta x} c^c \theta^{c-1}e^{-c \theta} \Gamma(c)^{-1}\text{d}\theta}\\ &=\frac{e^{-x}}{e^{-x}+c^c (c+x)^{-c-1} \Gamma(c+1)\Gamma(c)^{-1}}\\ &=\frac{e^{-x}}{e^{-x}+c^{c+1} (c+x)^{-c-1}}\\ &=\frac{e^{-x}}{e^{-x}+ (1+x/c)^{-c-1}}\\ \end{align*} which converges to $$\frac{e^{-x}}{e^{-x}+e^{-x}}=\frac{1}{2}$$ when $c$ goes to infinity.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.