2
$\begingroup$

Please is it ever possible for the prior distribution to contain more information about parameter(s) than the posterior distribution? If yes, when can that occur? Is it the same concept as the posterior being "diffuse with respect to the prior"? I am reading the paper on A Comparative Review of Dimension Reduction Methods in Approximate Bayesian Computation by Blum et al. (2013) and I came across the concept of a diffuse posterior on page 6. I would appreciate an explanation.

Thanks.

$\endgroup$
  • 2
    $\begingroup$ "More information" in what sense? $\endgroup$ – Tim Jan 31 at 16:17
  • $\begingroup$ I mean more information about the true value of the parameter(s) $\endgroup$ – emma Jan 31 at 16:19
  • $\begingroup$ IIRC there is a theorem that guarantees that the posterior has smaller variance than the priors, hence we would be more certain about the parameters after conditioning. $\endgroup$ – Demetri Pananos Jan 31 at 16:20
  • 2
    $\begingroup$ See stats.stackexchange.com/questions/322990/…. The answers there give two examples: dogmatic priors and unidentified parameters. $\endgroup$ – Christoph Hanck Jan 31 at 16:31
  • $\begingroup$ @Xi'an Yes, I think that is what I mean: The prior has higher variance than the posterior (save pathological examples where the prior is a point or something like that). I can't recall if it is actually a theorem or not, but I recall reading that somewhere. No guarantees however $\endgroup$ – Demetri Pananos Jan 31 at 17:19
3
$\begingroup$

It is possible for the posterior variance to be strictly greater than the prior variance. For example, suppose that $X\sim Bernoulli(p)$ with a prior $p\sim Beta(1, 10)$ and you observe $X=1$. Then, since we are using a conjugate prior, the posterior is a $Beta(2, 10)$.

The prior variance is $\frac{10}{11^2\cdot 12}\approx 0.0069$. The posterior variance is $\frac{20}{12^2\cdot 13}\approx 0.0107$. Here are plots of the prior and posterior distributions.

Prior and posterior densities

An intuition as to why this happens: the prior is skewed towards values of $p$ close to 0, but the likelihood favours values of $p$ close to 1. The posterior distribution is torn between the two and so has a larger variance.

$\endgroup$
0
$\begingroup$

The complete quote from the paper (which should have been included in the text of the question to make it self-contained) is

Since entropy measures information and a lack of randomness (Shannon, 1948), the authors propose minimizing the entropy of the approximate posterior, $p_\text{ABC}(θ|s^\text{obs})$, over subsets of the summary statistics, $s$, as a proxy for determining maximal information about a parameter of interest. High entropy results from a diffuse posterior sample, whereas low entropy is obtained from a posterior which is more precise in nature.

which does not mention the posterior being more diffuse than the prior. Now, as discussed in the comments, the posterior variance is on average smaller than the prior variance $$\text{var}(\theta)=\mathbb{E}[\text{var}(\theta|X)]+\text{var}(\mathbb{E}[\theta|X])$$ but it does not mean that $$\text{var}(\theta)\ge\text{var}(\theta|X=x)$$ for all realisations $x$ of $X$, as exemplified in Robin's answer.

$\endgroup$
  • $\begingroup$ Thanks for your answer. The particular statement in the paper which motivated my question is on page 6 : "An example of when this does not occur is when the posterior distribution is diffuse with respect to the prior-for instance, if an overly precise prior is located in the distributional tails of the posterior". So, I guess a follow-up question would be "What does it mean for a distribution to be diffuse with respect to another"? $\endgroup$ – emma Feb 2 at 16:06
  • $\begingroup$ This is not a standard term in (Bayesian) statistics. $\endgroup$ – Xi'an Feb 2 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.