9
$\begingroup$

A Bayes factor is defined in Bayesian testing of hypothesis and Bayesian model selection by the ratio of two marginal likelihoods: given an iid sample $(x_1,\ldots,x_n)$ and respective sampling densities $f_1(x|\theta)$ and $f_2(x|\eta)$, with corresponding priors $\pi_1$ and $\pi_2$, the Bayes factor for comparing the two models is $$\mathfrak{B}_{12}(x_1,\ldots,x_n)\stackrel{\text{def}}{=}\frac{m_1(x_1,\ldots,x_n)}{m_2(x_1,\ldots,x_n)}\stackrel{\text{def}}{=}\frac{\int \prod_{i=1}^n f_1(x_i|\theta)\pi_1(\text{d}\theta)}{\int \prod_{i=1}^n f_2(x_i|\eta)\pi_2(\text{d}\eta)}$$ A book I am currently reviewing has the strange affirmation that the above Bayes factor $\mathfrak{B}_{12}(x_1,\ldots,x_n)$ is "formed by multiplying the individual ones [Bayes factors] together" (p.118). This formally correct if one uses the decomposition \begin{align*}\mathfrak{B}_{12}(x_1,\ldots,x_n)&=\frac{m_1(x_1,\ldots,x_n)}{m_2(x_1,\ldots,x_n)}\\&=\frac{m_1(x_n|x_1,\ldots,x_{n-1})}{m_2(x_n|x_1,\ldots,x_{n-1})}\times \frac{m_1(x_{n-1}|x_{n-2},\ldots,x_1)}{m_2(x_{n-1}|x_{n-2},\ldots,x_1)}\times\cdots\\&\qquad\cdots\times\frac{m_1(x_1)}{m_2(x_1)}\end{align*} but I see not computational advantage in this decomposition as the update by$$\frac{m_1(x_n|x_1,\ldots,x_{n-1})}{m_2(x_n|x_1,\ldots,x_{n-1})}$$requires the same computational effort as the original computation of$$\frac{m_1(x_1,\ldots,x_n)}{m_2(x_1,\ldots,x_n)}$$outside artificial toy examples.

Question: Is there a generic and computationally efficient way of updating the Bayes factor from $\mathfrak{B}_{12}(x_1,\ldots,x_n)$ to $\mathfrak{B}_{12}(x_1,\ldots,x_{n+1})$ that does not require recomputing the entire marginals $m_1(x_1,\ldots,x_n)$ and $m_2(x_1,\ldots,x_n)$?

My intuition is that, besides particle filters, which indeed proceed along estimating the Bayes factors $\mathfrak{B}_{12}(x_1,\ldots,x_n)$ one new observation at a time, there is no natural way of answering this question.

$\endgroup$
  • 1
    $\begingroup$ It doesn't seem clear to me that the wording implies necessarily sequential factorization, as the observations are iid. During grad school, a professor mentioned that the product implies that one could use asymptotic approximations for Bayesian analyses but strangely this had not caught on (sarcasm). Perhaps the book could be hinting at that? $\endgroup$ – Cliff AB Sep 20 '15 at 16:53
  • $\begingroup$ @CliffAB: Yes, you could rewrite the likelihood as an average of individual terms, converging to a Kullback-Leibler distance from the true distribution. But I do not think this is the case, even though the book is unclear enough to keep all options open. $\endgroup$ – Xi'an Sep 20 '15 at 19:05
  • 1
    $\begingroup$ I believe there is a typo in the second displayed equation: should it be $m_1(x_{n-1}|x_{n-1}, \ldots, x_1)$ in the second factor on the second line? $\endgroup$ – jochen Apr 1 '16 at 19:38
2
$\begingroup$

Presumably the purpose of a recursive equation for Bayes factor would be when you have already calculated the Bayes factor for $n$ data points, and you want to be able to update this with one additional data point. It does seem that it is possible to do this without recomputing the marginals of the previous data vector, so long as the form of the posterior function $\pi_n$ is known. Assuming we know the form of this function (and assuming IID data as in your question), the predictive density can be written as:

$$\begin{equation} \begin{aligned} m(x_{n+1} | x_1,...,x_n) &= \int \limits_\Theta f(x_{n+1}|\theta) \pi_n(d \theta | x_1,...,x_n). \\[6pt] \end{aligned} \end{equation}$$

Hence, you have:

$$\begin{equation} \begin{aligned} m(x_1,...,x_{n+1}) &= m(x_1,...,x_n) \int \limits_\Theta f(x_{n+1}|\theta) \pi_n(d \theta | x_1,...,x_n). \\[6pt] \end{aligned} \end{equation}$$

Comparing two model classes via Bayes factor, we then get the recursive equation:

$$\begin{equation} \begin{aligned} \mathfrak{B}_{12}(x_1,...,x_{n+1}) &= \mathfrak{B}_{12}(x_1,...,x_{n}) \cdot \frac{\int _{\Theta_1} f(x_{n+1}|\theta) \pi_{1,n}(d \theta | x_1,...,x_n)}{\int _{\Theta_2} f(x_{n+1}|\theta) \pi_{2,n}(d \theta | x_1,...,x_n)}. \\[6pt] \end{aligned} \end{equation}$$

This still involves integration over the parameter range, so I agree with your view that there does not appear to be any computational advantage over just recomputing the Bayes factor via the initial formula you give. Nevertheless, you can see that this does not require you to recompute the marginals for the previous data vector. (Instead we compute the predictive densities of the new data point conditional on previous data, under each of the model classes.) Like you, I don't really see any computational advantage of this, but I suppose it gives you another formula for updating the Bayes factor.

$\endgroup$
  • $\begingroup$ Thank you. It is true that the marginals do not need to be re-computed, stricto sensu, but the amount of computation seems to be the same, as you remark. $\endgroup$ – Xi'an Jan 8 at 10:26
  • $\begingroup$ The only advantage I can think of is that since we are now only integrating over a single density (rather than the product of $n$ densities), the integrand will be less volatile, and so this latter formula might make it easier to avoid underflow problems in computation. That is all a big maybe though. $\endgroup$ – Reinstate Monica Jan 8 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.