7
$\begingroup$

Imagine a 1-dimensional Bayesian update, with a prior $P(\Theta = \theta)$ and a likelihood function for evidence $D$ written as $\mathcal{L}_D: \theta \mapsto f(\theta)$.

The posterior PDF $P(\Theta=\theta \mid D) \propto P(\Theta=\theta) \times f(\theta)$.

Suppose we now consider evidence $D'$ with a likelihood function shifted by $x$, i.e. $\mathcal{L}_{D'}: \theta \mapsto f(\theta-x) $.

For concreteness say $x>0$, so we're shifting the likelihood function to the right. My question is, what's a sufficient condition to imply that the expected value of the posterior is greater under $D'$ than under $D$? i.e. what's a sufficient condition for $\mathbb{E}[\Theta \mid D'] > \mathbb{E}[\Theta \mid D]$? Presumably it would be some condition on the shape of the prior PDF and likelihood function.

It's clear this isn't always the case. For example: enter image description here

Note that this counter-example is not just driven by the restricted domain of the uniform prior. You could just put arbitrarily thin tails on the ends of the uniform distribution I pictured (making it defined on $\mathbb{R}$ or anything else you want), without affecting the posterior expected value significantly.

I realise this question is very general, I would be interested in any sufficient conditions we can find, even if they are not the weakest achievable sufficient condition. However, a class of very strong sufficient conditions is not useful to me: conjugate priors. In the case of conjugate priors there is no need for the simulation I am doing (as the entire question is then answerable analytically).

One specific narrowing that would be particularly useful to me is: what if we add the restriction that the likelihood function is the normal likelihood function (the one that arises if the evidence is normally distributed)? This cannot be combined with the restriction that the prior should be normal, because then we are just back to conjugate priors.

The context is: I have a simulation where, in each iteration, we only care about whether the posterior expected value is greater than some bar, we needn't care what value it takes exactly. The answer to this question may help me make the simulation radically more efficient computationally.

If I am making an incorrect assumption or my notation is wrong, a correction is welcome.

$\endgroup$
10
  • $\begingroup$ It's probably worth thinking about the simpler, narrow case where L is the normal likelihood function first, before trying to move to a more general answer. $\endgroup$
    – tmkadamcz
    Commented Mar 16, 2022 at 20:41
  • $\begingroup$ Isn't $L(\theta-c)$ just a reparameterization of $\theta$, i.e., $\eta = \theta-c$? Why would the parameter (in the same likelihood) shift in light of new data? $\endgroup$
    – Daeyoung
    Commented Mar 16, 2022 at 21:01
  • $\begingroup$ Good question @DaeyoungLim. I think the case where the evidence is normally distributed is a case where such a shift makes sense. I don't know if there are other cases where this is a "sensible" way to describe new data (there are cases where it's not sensible, such as a binomial likelihood function -- but that tells little). If we can solve the normal case that will be a nice start and will help me get a grasp on the problem. $\endgroup$
    – tmkadamcz
    Commented Mar 16, 2022 at 21:08
  • $\begingroup$ I believe that makes no difference because $\theta$ already indicates the location of the data. If the data shifted, so will $\theta$. Reparamaterizing a location parameter by a scalar difference is a meaningless operation. Am I misconstruing your problem? $\endgroup$
    – Daeyoung
    Commented Mar 16, 2022 at 21:10
  • $\begingroup$ $\theta$ is the parameter about which we are conducting inference. $\endgroup$
    – tmkadamcz
    Commented Mar 16, 2022 at 21:12

3 Answers 3

2
$\begingroup$

Here's a very restrictive sufficient condition: both the prior and likelihood are normal.

If the likelihood is normal with known variance $\sigma^2$, and the prior is normal, then the posterior is normal. Say the likelihood is normal centred on $z$ with variance $\sigma^2$, and the prior has mean $\mu_0$ and variance $\sigma_0^2$, then the posterior has mean $\Big(\frac{1}{\sigma_0^2}+\frac{1}{\sigma^2}\Big)^{-1}\Big(\frac{\mu_0}{\sigma_0^2}+\frac{z}{\sigma^2}\Big)$, which is strictly increasing in $z$.

If this is too restrictive, you could consider other conjugate priors: https://en.wikipedia.org/wiki/Conjugate_prior.

$\endgroup$
1
  • $\begingroup$ Thanks! Yeah, this is correct, but any situation where there is a closed form expression for the posterior will not be useful to me. Otherwise I could do everything analytically without doing a simulation. I'm only interested in cases where the posterior has to be calculated numerically. Will clarify the question. $\endgroup$
    – tmkadamcz
    Commented Mar 16, 2022 at 20:52
2
$\begingroup$

It works for all priors in the case that $ \mathcal L $ is normal. The key property is that $ \mathcal L'(\theta) = -\theta \mathcal L(\theta) $.

To see this, start with expressing the expectation under the two hypotheses as

$$ \mathbb E[\theta \mid D] = \frac{\mathbb E[\theta \mathcal L_D(\theta)]}{\mathbb E[\mathcal L_D(\theta)]}, \, \, \mathbb E[\theta \mid D'] = \frac{\mathbb E[\theta \mathcal L_{D'}(\theta)]}{\mathbb E[\mathcal L_{D'}(\theta)]}, $$

where the expectations are taken over the prior distribution $ \mathbb P(\theta) $. Denote $ \mathcal L_D(\theta) = L(\theta) $ so that $ \mathcal L_{D'}(\theta) = L(\theta - x) $. Then what we have to show is that

$$ \frac{\mathbb E[\theta L(\theta)]}{\mathbb E[L(\theta)]} \leq \frac{\mathbb E[\theta L(\theta-x)]}{\mathbb E[L(\theta-x)]} = g(x) $$

or equivalently that the function $ g $ on the right hand side is increasing when seen as a function of $ x $. Differentiating with respect to $ x $ is fine because $ L $ and its derivatives decay very fast at infinity and the prior $ \mathbb P(\theta) $ is $ L^1 $ since it's a probability density. We get

$$ g'(x) = \frac{ -\mathbb E[\theta L'(\theta - x)] \mathbb E[L(\theta - x)] + \mathbb E[\theta L(\theta - x)] \mathbb E[L'(\theta - x)]}{\mathbb E[L(\theta - x)]^2} $$

and we want to show $ g' $ is always greater than or equal to zero.

Now we use the fact that $ L'(\theta) = -\theta L(\theta) $ - any positive proportional constant here is also fine, of course. Substituting directly to the numerator gives

$$ \frac{\mathbb E[\theta (\theta - x) L(\theta - x)]}{\mathbb E[L(\theta - x)]} - \frac{\mathbb E[\theta L(\theta - x)]}{\mathbb E[L(\theta - x)]} \frac{\mathbb E[(\theta - x) L(\theta - x)]}{\mathbb E[L(\theta - x)]} $$

The positivity of this expression is simply the Cauchy-Schwarz inequality for the probability measure on $ \theta $ whose density is

$$ \frac{\mathbb P(\theta) L(\theta - x)}{\int_{\mathbb R} \mathbb P(\theta) L(\theta - x)\, d \theta} $$

so we're done.

$\endgroup$
3
  • $\begingroup$ Hey Ege! Can we discuss this on this chat, which allows LaTeX? mathim.com/toms_math $\endgroup$
    – tmkadamcz
    Commented Apr 12, 2022 at 18:53
  • $\begingroup$ This looks great, except that in the question, I ended up calling $f$ the likelihood function to clarify things, and I hadn't seen you used $f$ for something else, which now makes this answer confusing. So it would be great to edit accordingly. $\endgroup$
    – tmkadamcz
    Commented Apr 12, 2022 at 19:01
  • $\begingroup$ @tmkadamcz Done. $\endgroup$
    – Ege Erdil
    Commented Apr 12, 2022 at 19:13
2
$\begingroup$

Thanks to everyone who contributed. Here are some results I've been able to find in the literature.

The strongest result is that the property holds for any prior when the likelihood function arises from a strongly unimodally distributed observation.

Andrews et al. 1972 (Biometrika, Vol. 59, No. 3 (Dec., 1972), pp. 693-695) use the same approach as Ege Erdil:

enter image description here

Ma 1997 (Statistics & Probability Letters 42 (1999) 33-37) generalises this to any likelihood function arising from an observation whose distribution has $\Theta$ as a location parameter and is strongly unimodally distributed. After noting that "a non-degenerate distribution $F$ is strongly unimodal if and only if it has a density $f$ that is log-concave within some open interval", he proves:

enter image description here

Here is the link to Mitchell 1994 (J. R. Statist. Soc. B (1994) 56, No.4, pp. 605-610).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.