Bayesian hypothesis testing and Bayes factors

Question

Supposing a parameter of interest is assumed to follow an exponential distribution with rate $\lambda$. Two competing prior hypotheses are formed such that the first prior is given by $\text{Ga}(\alpha, \beta)$ and the second prior is given by $\text{Ga}(\alpha+1, \beta)$. Furthermore, data is collected on the parameter such that $x_{1}, x_{2}, \dots, x_{n}$ represent $n$ observations of recorded data.

The individual responsible for forming these priors suggests a degree of belief such that $p(\lambda \sim \text{Ga}(\alpha, \beta)) = 0.75$ and $p(\lambda \sim \text{Ga}(\alpha+1, \beta)) = 0.25$.

From the above:

• How would a Bayes factor for the two prior distributions be formed?

Can this be achieved by calculating the ratio between the two prior distributions?

i.e. $$\text{Bayes} factor = \frac{p(x) \sim \text{Ga}(\alpha, \beta)}{p(x) \sim \text{Ga}(\alpha+1, \beta)} = \frac{\frac{\beta^{\alpha}}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}}{\frac{\beta^{\alpha+1}}{\Gamma(\alpha+1)}x^{\alpha}e^{-\beta x}}$$

If so, given that there are multiple observations ($x_{i}$), how should each observation be inserted into the distribution to give a composite distribution (is it the sum of $p(x) \sim \text{Ga}(\alpha, \beta)$ such that $\sum_{i=1}^{n} p(x) \sim \text{Ga}(\alpha, \beta)$)? In which case the Bayes factor is given by:

$$\frac{\sum_{i=1}^{n} p(x_{i}) \sim \text{Ga}(\alpha, \beta)}{\sum_{i=1}^{n} p(x_{i}) \sim \text{Ga}(\alpha+1, \beta)}$$

• How can an estimate of the observed data be combined with the model probabilities to achieve a posterior ratio of model probabilities?

Can this be achieved by getting the product of the prior and likelihood distributions?

$$p(\lambda|{\bf x}) = p(x|\lambda)p(\lambda)$$

If so, assuming there is only a collection of observations and no suggested likelihood, how can the correct likelihood be determined?

This a generic homework-like template question, as opposed to a specific exercise hence the absence of specific parameter and data values.

Answer 1

Let $p(\alpha,\beta|M_1)=\mathrm{Ga}(\alpha,\beta)$ be your first prior, and $p(\alpha,\beta|M_2)=\mathrm{Ga}(\alpha+1,\beta)$ be your second prior. Furthermore, let $D$ be your observed data.

Your posterior distribution can then be expressed as: $$p(\alpha,\beta|D,M_i) = \frac{L(D|\alpha,\beta)p(\alpha,\beta|M_i)}{p(D|M_i)}$$ where $p(D|M_i)=\int\int L(D|\alpha,\beta)p(\alpha,\beta|M_i) \,\mathrm{d}\alpha\mathrm{d}\beta$ and $i$ is either 1 or 2.

How would a Bayes factor for the two prior distributions be formed?

The Bayes factor is defined as $$K=\frac{p(D|M_1)}{p(D|M_2)}$$

Meaning, that the Bayes factor is not formed for the two prior distributions alone, but always in combination with the respective likelihood. Typically, in Bayesian analysis we compare different likelihood functions. However, comparing different prior assumptions also makes perfect sense in some cases.

A typical application would be to make robust predictions: $$p(\alpha,\beta|D) = p(\alpha,\beta|D,M_1)p(M_1|D) + p(\alpha,\beta|D,M_2)p(M_2|D) $$ with $$ p(M_i|D) = \frac{p(D|M_i) p(M_i)}{p(D|M_1) p(M_1)+p(D|M_2) p(M_2)} $$ Assuming that you are indifferent between $M_1$ and $M_2$ a priori, we have $p(M_i)=0.5$

How can an estimate of the observed data be combined with the model probabilities to achieve a posterior ratio of model probabilities?

I am not sure if I understand your question correctly, but in my opinion this is the Bayes factor. A value of $K > 1$ means that $M_1$ is more strongly supported by the data under consideration than $M_2$.

Answer 2

How would a Bayes factor for the two prior distributions be formed?

The Bayes factor $K$ for competing models $M_1$ and $M_2$ is the quotient of the data likelihoods of the two models. Applying this definition to the problem you describe gives:

$$ K = \frac{p(\mathbf{x} |M_1)}{p(\mathbf{x}|M_2)} = \frac{\int_{\lambda}{p(\mathbf{x}|\lambda)p(\lambda|M_1)}d\lambda}{\int_{\lambda}{p(\mathbf{x}|\lambda)p(\lambda|M_2)}d\lambda} $$

Stating the densities explicitly gives:

$$ K = \frac {\int_{\lambda} {(\prod_{i=1}^n{\lambda e^{-\lambda x_i})} \frac{\beta^{\alpha}}{\Gamma(\alpha)}{\lambda}^{\alpha-1}e^{-\beta \lambda}}} {\int_{\lambda} {(\prod_{i=1}^n{\lambda e^{-\lambda x_i})} \frac{\beta^{\alpha + 1}}{\Gamma(\alpha+1)}{\lambda}^{\alpha}e^{-\beta \lambda}}} = \frac {\frac{\beta^{\alpha}}{\Gamma(\alpha)} \int_\lambda \lambda^{n+\alpha-1}e^{-\lambda(\beta + n\bar{x})} } {\frac{\beta^{\alpha+1}}{\Gamma(\alpha + 1)} \int_\lambda \lambda^{n+\alpha}e^{-\lambda(\beta + n\bar{x})} } $$

(Using $n\bar{x} = \sum_{i=1}^n{x_i}$ for convenience.) We recognize the integrand as the kernels of gamma distributions Gamma($n+\alpha, \beta + n\bar{x}$) and Gamma($n+\alpha + 1, \beta + n\bar{x}$). This tells us that each integrates to the reciprocal of its normalizing constant, giving:

$$ K = \frac { \frac{\beta^{\alpha}}{\Gamma(\alpha)} \frac{\Gamma(\alpha+n)}{(\beta + n\bar{x})^{\alpha+n}} } { \frac{\beta^{\alpha+1}}{\Gamma(\alpha+1)} \frac{\Gamma(\alpha+n+1)}{(\beta + n\bar{x})^{\alpha+n+1}} } = \frac {\beta^\alpha\Gamma(\alpha+1)\Gamma(\alpha+n)(\beta + n\bar{x})^{\alpha+n+1}} {\beta^{\alpha+1}\Gamma(\alpha)\Gamma(\alpha+n+1)(\beta + n\bar{x})^{\alpha+n}} $$

By the $\Gamma(\alpha+1) = \alpha\Gamma(\alpha)$ property of the Gamma function:

$$ K = \frac {\alpha\Gamma(\alpha)\Gamma(\alpha+n)(\beta + n\bar{x})} {\beta\Gamma(\alpha)(\alpha+n)\Gamma(\alpha + n)} = \frac {\alpha(\beta + n\bar{x})} {\beta(\alpha + n)} = \frac {\alpha\beta + \alpha n \bar{x}} {\alpha\beta + \beta n } $$

...given that there are multiple observations $x_i$, how should each observation be inserted into the distribution to give a composite distribution[?]

This is accomplished above by the product in my second set of equations. Assuming each $x_i$ is independently drawn from the exponential, the joint density $p(\bf{x})$ is the product of the individual densities.

How can an estimate of the observed data be combined with the model probabilities to achieve a posterior ratio of model probabilities?

Solving this means finding the posterior model odds using the priors for $M_1, M_2$:

$$ \frac{p(\mathbf{x} |M_1)p(M_1)}{p(\mathbf{x}|M_2)p(M_2)} = K\frac{p(M_1)}{p(M_2)} = K\frac{\frac{3}{4}}{\frac{1}{4}} = 3K $$

In other words, the Bayes factor is equivalent to the posterior model odds when competing models are assumed equally likely.