2
$\begingroup$

Please is it ever possible for the prior distribution to contain more information about parameter(s) than the posterior distribution? If yes, when can that occur? Is it the same concept as the posterior being "diffuse with respect to the prior"? I am reading the paper on A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation by Blum et al. (2013) and I came across the concept of a diffuse posterior on page 6. I would appreciate an explanation.

Thanks.

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ "More information" in what sense? $\endgroup$
    – Tim
    Commented Jan 31, 2019 at 16:17
  • $\begingroup$ I mean more information about the true value of the parameter(s) $\endgroup$
    – emma
    Commented Jan 31, 2019 at 16:19
  • $\begingroup$ IIRC there is a theorem that guarantees that the posterior has smaller variance than the priors, hence we would be more certain about the parameters after conditioning. $\endgroup$
    – Demetri Pananos
    Commented Jan 31, 2019 at 16:20
  • 2
    $\begingroup$ See stats.stackexchange.com/questions/322990/…. The answers there give two examples: dogmatic priors and unidentified parameters. $\endgroup$
    – Christoph Hanck
    Commented Jan 31, 2019 at 16:31
  • 1
    $\begingroup$ @kjetilbhalvorsen Since writing this I've learned that this is generally false. $\endgroup$
    – Demetri Pananos
    Commented Aug 2, 2020 at 22:03

2 Answers 2

Reset to default
4
$\begingroup$

It is possible for the posterior variance to be strictly greater than the prior variance. For example, suppose that $X\sim Bernoulli(p)$ with a prior $p\sim Beta(1, 10)$ and you observe $X=1$. Then, since we are using a conjugate prior, the posterior is a $Beta(2, 10)$.

The prior variance is $\frac{10}{11^2\cdot 12}\approx 0.0069$. The posterior variance is $\frac{20}{12^2\cdot 13}\approx 0.0107$. Here are plots of the prior and posterior distributions.

Prior and posterior densities

An intuition as to why this happens: the prior is skewed towards values of $p$ close to 0, but the likelihood favours values of $p$ close to 1. The posterior distribution is torn between the two and so has a larger variance.

Improve this answer
$\endgroup$
1
$\begingroup$

The complete quote from the paper (which should have been included in the text of the question to make it self-contained) is

Since entropy measures information and a lack of randomness (Shannon, 1948), the authors propose minimizing the entropy of the approximate posterior, $p_\text{ABC}(θ|s^\text{obs})$, over subsets of the summary statistics, $s$, as a proxy for determining maximal information about a parameter of interest. High entropy results from a diffuse posterior sample, whereas low entropy is obtained from a posterior which is more precise in nature.

which does not mention the posterior being more diffuse than the prior. Now, as discussed in the comments, the posterior variance is on average smaller than the prior variance $$\text{var}(\theta)=\mathbb{E}[\text{var}(\theta|X)]+\text{var}(\mathbb{E}[\theta|X])$$ but it does not mean that $$\text{var}(\theta)\ge\text{var}(\theta|X=x)$$ for all realisations $x$ of $X$, as exemplified in Robin's answer.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer. The particular statement in the paper which motivated my question is on page 6 : "An example of when this does not occur is when the posterior distribution is diffuse with respect to the prior-for instance, if an overly precise prior is located in the distributional tails of the posterior". So, I guess a follow-up question would be "What does it mean for a distribution to be diffuse with respect to another"? $\endgroup$
    – emma
    Commented Feb 2, 2019 at 16:06
  • $\begingroup$ This is not a standard term in (Bayesian) statistics. $\endgroup$
    – Xi'an
    Commented Feb 2, 2019 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.