12
$\begingroup$

(This question is inspired by this comment from Xi'an.)

It is well known that if the prior distribution $\pi(\theta)$ is proper and the likelihood $L(\theta | x)$ is well-defined, then the posterior distribution $\pi(\theta|x)\propto \pi(\theta) L(\theta|x)$ is proper almost surely.

In some cases, we use instead a tempered or exponentiated likelihood, leading to a pseudo-posterior

$$\tilde\pi(\theta|x)\propto \pi(\theta) L(\theta|x)^\alpha$$ for some $\alpha>0$ (for example, this can have computational advantages).

In this setting, is it possible to have a proper prior but an improper pseudo-posterior?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Actually, a few minutes later, I would consider it unlikely since the divergence of the prior x likelihood product is reduced when considering the prior x likelihood^ α product... Any tern going to infinity is going there more slowly! And terms going to zero more slowly are controlled by the proper prior. My bet is thus that this is impossible. (warning: I have been known to be wrong!) $\endgroup$
    – Xi'an
    Oct 9, 2018 at 8:58
  • 1
    $\begingroup$ Possibly useful in seeking a counterexample when $\alpha > 1$: Markov's inequality tells us that $$\mathbf{E}_{\theta \sim \pi} \left[ L(x| \theta)^\alpha \right] \geqslant t^\alpha \mathbf{P}_{\theta \sim \pi} (L(x| \theta) > t) \\ \implies \mathbf{E}_{\theta \sim \pi} \left[ L(x| \theta)^\alpha \right] \geqslant \sup_{t > 0} t^\alpha \mathbf{P}_{\theta \sim \pi} (L(x| \theta) > t) $$ So if you can find a case where $L(x| \theta)$ has polynomial tails, then you can possibly construct an improper pseudo-posterior. $\endgroup$
    – πr8
    Oct 14, 2018 at 10:54
  • $\begingroup$ Would this argument also work for $\alpha < 1$? Also, is there a way to prove that a likelihood constructed in this fashion would be proper? $\endgroup$
    – adityar
    Oct 14, 2018 at 11:27
  • 1
    $\begingroup$ Actually, for $\alpha =1$, since we know that $\mathbf{E}_{\pi} [L (x | \theta)] < \infty$, the supremum on the RHS is always finite, and for $\alpha < 1$, one uses your Jensen argument to make the same deduction. So the argument fails in that respect. A small remark that this argument requires an unbounded likelihood $L$ to succeed, i.e. $\mathbf{P}_\pi (L (x | \theta) > t) > 0$ for all $t$. $\endgroup$
    – πr8
    Oct 14, 2018 at 11:48
  • 1
    $\begingroup$ True, for $\alpha = 1$, you cannot construct one, good point! I must say, I'd be fascinated to see an example of an unbounded likelihood! Perhaps a beta posterior would be a result of an unbounded likelihood. $\endgroup$
    – adityar
    Oct 14, 2018 at 12:42

2 Answers 2

Reset to default
8
$\begingroup$

For $\alpha \leq 1$, perhaps this is an argument to show that it is impossible to construct such a posterior?

We'd like to find out if it's possible for $\int \tilde \pi(\theta|x)d\theta = \infty$.

On the RHS:

$$ \int \pi(\theta) L^{\alpha}(\theta|x) d\theta = E_{\theta}(L^{\alpha}(\theta|x))$$

If $\alpha \leq 1$, $x^{\alpha}$ is a concave function, so by the Jensen inequality:

$$ E_{\theta}(L^{\alpha}(\theta|x)) \leq E^{\alpha}_{\theta}(L(\theta|x)) = m(x)^\alpha < \infty $$

... where $m(x)$ as Xi'an pointed out, is the normalising constant (the evidence).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Neat, thanks. I like that you are using the fact that for $\alpha=1$ the posterior is proper. $\endgroup$
    – Robin Ryder
    Oct 9, 2018 at 11:15
  • 2
    $\begingroup$ Just wanted to say that this discussion ended up leading to a research paper: onlinelibrary.wiley.com/doi/10.1002/sim.9124, which I think is a very nice outcome! $\endgroup$
    – Luiz Max Carvalho
    Sep 5, 2021 at 19:40
1
$\begingroup$

It's possible to use the result in @InfProbSciX's answer to prove the result in general. Rewrite $L(\theta\mid x)^\alpha\pi(\theta)$ as $$L(\theta\mid x)^{\alpha-1}L(\theta\mid x)\pi(\theta).$$ If $1 \leq \alpha \leq 2$, we have the Jensen's inequality case above, since we know that $L(x|\theta)\pi(\theta)$ is normalisable. Similarly, if $2 \leq \alpha \leq 3$, we can write $$ L(x|\theta)^{\alpha-p} L(x|\theta)^p\pi(\theta),$$ with $1 \leq p \leq 2$, again falling into the same case, since we know that $L(x|\theta)^{p}\pi(\theta)$ is normalisable. Now one can use (strong) induction to show the case in general.

Old comments

Not sure if this is super useful, but since I can't comment I will leave this in an answer. In addition to @InfProbSciX's excellent remark about $\alpha \leq 1$, if one makes the further assumption that $L(\theta \mid x) \in L^p$, then it is impossible to have a proper prior but an improper pseudo-posterior for $ 1 < \alpha \leq p$. For instance, if we know that the second ($p$-th) moment of $L(\theta \mid x)$ exists, we know it is in $L^2$ ($L^p$) and hence the pseudo-posterior will proper for $0 \leq \alpha \leq 2$. Section 1 in these notes goes into a bit more detail, but unfortunately it is not clear how broad the class of, say, $L^{10}$ pdfs is. I apologise if I'm speaking out of turn here, I really wanted to leave this as a comment.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ You're right, if the likelihood function $L(\theta|x)$ is within the space $L^p(\bf {\pi_\theta})$ - i.e. the $L^p$ space w.r.t. the measure induced by the prior, then the posterior will be proper for $1 \leq \alpha \leq p$. I'm totally guessing here, but I think that the space would encompass most likelihoods we can think of - I think I might have read a proof ages ago that says that if $f$ is Riemann integrable, then its positive powers are as well. $f^n, n \in \mathbb Z^+$ is integrable though. Theorem 1.26 for reference $\endgroup$
    – adityar
    Oct 9, 2018 at 15:44
  • $\begingroup$ @InfProbSciX, I think there might be a complete proof lurking in the shadows here. I take from your answer that $\alpha$ can be negative. If that is correct, then we can show that for any $p > 1$ the pseudo-likelihood will be integrable because reciprocals of integrable functions are integrable. And if the likelihood is integrable, I argue that the posterior will be integrable because the prior is bounded, and the product of an integrable and a bounded function is integrable (math.stackexchange.com/a/56008/271610). Let me know what you think. $\endgroup$
    – Luiz Max Carvalho
    Oct 12, 2018 at 10:19
  • 1
    $\begingroup$ I think that you can disregard the case where $\alpha < 0$, as the question explicitly assumes otherwise. The integrability of $L^{\alpha}$ for any general case needs to be shown. Also, I'm not sure if the prior is always bounded, for example, the density of a $Beta(0.5, 0.5)$ wouldn't be. $\endgroup$
    – adityar
    Oct 12, 2018 at 10:42
  • $\begingroup$ @InfProbSciX, what I meant was that even if $\alpha < 0$ is not in the question, if your proof holds for that condition also, then we could show integrability for $\alpha > 1$ by leveraging the fact that if $f$ is integrable then so is $1/f$. As you say, all of that is nil if the prior is unbounded. We can try to bound the likelihood instead and It seems to me that any likelihood one would use in MLE would have to be either bounded or strongly concave (en.wikipedia.org/wiki/Maximum_likelihood_estimation#Properties) both of which can be used to build a general proof. Any thoughts? $\endgroup$
    – Luiz Max Carvalho
    Oct 20, 2018 at 19:15
  • $\begingroup$ Sorry, I missed that, yeah that looks like it'd make an interesting attempt! $\endgroup$
    – adityar
    Oct 23, 2018 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.