What is the "effective sample size" of the prior in Bayesian statistics?

Question

In Bayesian statistic, what is the mathematical definition of "effective sample size" of the prior? Could you provide what the "effective sample size" is for the well known classes of conjugate priors? Does this concept generalize to non-conjugate models? Why is the idea of "effective sample size" of the prior it important?

Edit for the bounty: this question is very important and deserves a complete, canonical answer with more examples and proper explanations (trying to address the questions listed above).

Answer 1

Here is an example with a beta prior distribution and a binomial likelihood.

Suppose the prior distribution of the heads probability $\theta$ of a coin is $\mathsf{Beta}(10,10)$ and that $n = 100$ tosses of the coin yield $x = 47$ Heads. Then the posterior distribution of $\theta$ is $\mathsf{Beta}(10 + x, 10 + 100 - x) \equiv \mathsf{Beta}(57, 63).$

This results from Bayes' Theorem, multiplying prior $f(\theta)$ by likelihood $g(x|\theta)$ to get posterior $h(\theta|x):$

$$f(\theta)\times g(x|\theta) \propto \theta^{10-1}(1-\theta)^{10-1} \times \theta^{x}(1-\theta)^{n-x}\\ \propto h(\theta|x) \propto \theta^{(10+x)-1}(1-\theta)^{(10+100-x)-1} \propto \theta^{57-1}(1-\theta)^{63-1}.$$

One could say that the prior distribution is 'effectively' equivalent to advance knowledge of $20$ tosses of the coin yielding 10 heads.

Note: In the displayed relationship for Bayes' Theorem, use of the symbol $\propto$ (read "proportional to"), instead of $=,$ recognizes that we are showing the kernels (density functions without their norming constants) of the prior, likelihood, and posterior. Because the prior and likelihood are 'conjugate' (mathematically compatible), we can recognize the expression on the right as the kernel of $\mathsf{Beta}(57, 63).$

Answer 2

Prior effective sample size (ESS) doesn't have a single definition. It's a heuristic to understand the influence of the prior on the posterior parameters. Prior ESS tells you that your choice of prior is comparable to having an additional $n_{E}$ data points.

It is straightforward to demonstrate prior ESS with conjugate priors. It is more complicated when you have non-conjugate priors.

Conjugate Priors

Beta-Binomial Example

Say you having a binomial random variable, $Y$ and you want to estimate the probability of success, $\theta$. You observe $y$ successes and $n-y$ failures. Assume a $Beta(\alpha, \beta)$ for $\theta$. $$ Y \sim Binom(n, \theta),$$ $$\theta \sim Beta(\alpha, \beta), \text{ and }$$ $$ \theta | Y \sim Beta(\alpha + y, \beta + n - y)$$

If the prior contributed no information to the posterior, we would use only the data to inform the posterior: $\theta | y \sim Beta(y, n - y)$. We can compare the parameters of this noninformative posterior to the general form of the posterior to see how the prior adds information to the likelihood. The $Beta(\alpha, \beta)$ prior is equivalent to observing an additional $\alpha$ heads and an additional $\beta$ tails. So the prior effective sample size is $\alpha + \beta$.

Non-conjugate case

When the prior and the likelihood aren't conjugate, we can't see exactly how the prior parameters interact with the likelihood to make the posterior parameters. Morita et all (2008) generalize the concept from conjugate distributions to distributions in general. I'll give the concept, but you can reference that paper for all the details.

You find the prior ESS with the algoritm:

1. Choose your prior distribution. Calculate the information matrix of the prior parameter.
2. Identify a non-informative alternative to (1). Ex: $\theta \sim Normal(\mu = 0, \sigma^2 = 10e5)$
3. Take a sequence from $m = 0, \dots, M$ and get the posterior for each prior (2) with a dataset of size $m$. Calculate the information matrix.
4. The prior ESS is the same size, $m_o$, which minimizes the distance of between the prior information from (1) and posterior information from (3).