14
$\begingroup$

In Bayesian statistic, what is the mathematical definition of "effective sample size" of the prior? Could you provide what the "effective sample size" is for the well known classes of conjugate priors? Does this concept generalize to non-conjugate models? Why is the idea of "effective sample size" of the prior it important?

Edit for the bounty: this question is very important and deserves a complete, canonical answer with more examples and proper explanations (trying to address the questions listed above).

$\endgroup$
1
  • $\begingroup$ An example of the use of this concept: this blog post makes the argument that the difference between a Beta(1, 1) prior and a Beta(0.7, 1) is negligible when the actual sample size is 200. $\endgroup$
    – fblundun
    Mar 23, 2021 at 22:55

2 Answers 2

11
$\begingroup$

Here is an example with a beta prior distribution and a binomial likelihood.

Suppose the prior distribution of the heads probability $\theta$ of a coin is $\mathsf{Beta}(10,10)$ and that $n = 100$ tosses of the coin yield $x = 47$ Heads. Then the posterior distribution of $\theta$ is $\mathsf{Beta}(10 + x, 10 + 100 - x) \equiv \mathsf{Beta}(57, 63).$

This results from Bayes' Theorem, multiplying prior $f(\theta)$ by likelihood $g(x|\theta)$ to get posterior $h(\theta|x):$

$$f(\theta)\times g(x|\theta) \propto \theta^{10-1}(1-\theta)^{10-1} \times \theta^{x}(1-\theta)^{n-x}\\ \propto h(\theta|x) \propto \theta^{(10+x)-1}(1-\theta)^{(10+100-x)-1} \propto \theta^{57-1}(1-\theta)^{63-1}.$$

One could say that the prior distribution is 'effectively' equivalent to advance knowledge of $20$ tosses of the coin yielding 10 heads.

Note: In the displayed relationship for Bayes' Theorem, use of the symbol $\propto$ (read "proportional to"), instead of $=,$ recognizes that we are showing the kernels (density functions without their norming constants) of the prior, likelihood, and posterior. Because the prior and likelihood are 'conjugate' (mathematically compatible), we can recognize the expression on the right as the kernel of $\mathsf{Beta}(57, 63).$

$\endgroup$
13
  • 1
    $\begingroup$ Great, how can this generalize in general case when you cannot find conjugacy? $\endgroup$
    – Good Luck
    Mar 23, 2021 at 8:55
  • 2
    $\begingroup$ Not all Bayesian statisticians find the concept of 'effective sample size' useful. I have not seen it used in general contexts, where prior and posterior are not in the same distribution family. // If trying to choose a reasonable prior in the coin example, I might say the coin looks fair (not bent, circular, no nodules, etc.) so maybe $E(\theta) \approx 1/2.$ Guess maybe $P(1/3 < \theta < 2/3) \ge .9.$ That matches $\mathsf{Beta}(10,10)$ pretty well, so that'll be my prior. Unlikely I imagined 20 hypothetical coin tosses with 10 Heads. $\endgroup$
    – BruceET
    Mar 23, 2021 at 16:18
  • $\begingroup$ Maybe someone else has a different view of this and knows of a general math formulation of 'effective sample size'. Seems to me it's often used vaguely as a way to try to make the idea of prior distribution seem reasonable to a non-bayesian. $\endgroup$
    – BruceET
    Mar 23, 2021 at 16:22
  • 4
    $\begingroup$ Not writing a textbook here, just trying to illustrate a concept that some find useful. $\endgroup$
    – BruceET
    Mar 23, 2021 at 20:37
  • 1
    $\begingroup$ The internet does seem to agree that the effective sample size is the sum of the shape parameters. Interesting that the ESS for the uniform prior is 2 rather than 0. It seems like this definition of the ESS uses the improper Haldane prior as the baseline. $\endgroup$
    – fblundun
    Mar 23, 2021 at 23:41
4
$\begingroup$

Prior effective sample size (ESS) doesn't have a single definition. It's a heuristic to understand the influence of the prior on the posterior parameters. Prior ESS tells you that your choice of prior is comparable to having an additional $n_{E}$ data points.

It is straightforward to demonstrate prior ESS with conjugate priors. It is more complicated when you have non-conjugate priors.

Conjugate Priors

Beta-Binomial Example

Say you having a binomial random variable, $Y$ and you want to estimate the probability of success, $\theta$. You observe $y$ successes and $n-y$ failures. Assume a $Beta(\alpha, \beta)$ for $\theta$. $$ Y \sim Binom(n, \theta),$$ $$\theta \sim Beta(\alpha, \beta), \text{ and }$$ $$ \theta | Y \sim Beta(\alpha + y, \beta + n - y)$$

If the prior contributed no information to the posterior, we would use only the data to inform the posterior: $\theta | y \sim Beta(y, n - y)$. If we compare the parameters of this distribution to the general case, we see that $\beta$ is augmenting $n$ in the posterior. The prior effective sample size is $\beta$.

Non - conjugate case

When the prior and the likelihood aren't conjugate, we can't see exactly how the prior parameters interact with the likelihood to make the posterior parameters. Morita et all (2008) generalize the concept from conjugate distributions to distributions in general. I'll give the concept, but you can reference that paper for all the details.

You find the prior ESS with the algoritm:

  1. Choose your prior distribution. Calculate the information matrix of the prior parameter.
  2. Identify a non-informative alternative to (1). Ex: $\theta \sim Normal(\mu = 0, \sigma^2 = 10e5)$
  3. Take a sequence from $m = 0, \dots, M$ and get the posterior for each prior (2) with a dataset of size $m$. Calculate the information matrix.
  4. The prior ESS is the same size, $m_o$, which minimizes the distance of between the prior information from (1) and posterior information from (3).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy