5
$\begingroup$

Suppose that we have data, $D$, and two parameters we want to learn about, $\theta_1, \theta_2$. We will usually put priors on $\theta_1, \theta_2$, then have the expression:

$$ p(\theta_1, \theta_2\mid D) \propto p(D\mid \theta_1, \theta_2) p(\theta_1) p(\theta_2) $$

I am wondering why most set-ups assume that the priors above are independent. What happens if we do not have it?

$\endgroup$
  • 1
    $\begingroup$ Thank you for asking this. I am new to Bayesian inference and could not find/construct a proof of the biggest draw to it - that the posterior is decreasingly dependent on the priors with more data. $\endgroup$ – Debanjan Basu Apr 29 '18 at 13:45
  • $\begingroup$ The short answer is "nothing happens". People make the assumption of independence in situations where it's plausible, or where its approximately true, for simplicity, or if they have a lot of data relative to the prior information (in which case the prior is likely to have little effect on the final results anyway.) But there's no harm in using a joint prior, I've done so myself in quite a few situations. $\endgroup$ – jbowman Apr 30 '18 at 17:46
  • $\begingroup$ This is mostly for convenience as it is more difficult to create a joint prior. The exception being the Jeffreys prior which makes parameters dependent through the determinant of the Fisher information matrix. $\endgroup$ – Xi'an May 2 '18 at 19:40
  • $\begingroup$ @DebanjanBasu maybe I'm missing something, but I don't see this question being related to how the posterior behaves with more data $\endgroup$ – Juho Kokkala May 4 '18 at 16:13
  • $\begingroup$ @JuhoKokkala this was the intuition I was given about Bayes' Theorem. This discussion is in the same vein. I think it does provide a useful justification for using "simple" priors - like the assumption of independence. $\endgroup$ – Debanjan Basu May 22 '18 at 18:01
4
+25
$\begingroup$

Your expression is only correct when you assume independent priors. Otherwise, the expression would become

$$ p(\theta_1, \theta_2 | D) \propto p(D|\theta_1, \theta_2) p(\theta_1, \theta_2) $$

In this expression, you may need further assumptions to work with $p(\theta_1, \theta_2)$ or you can work with it as is. If you assume independence you get your expression again. From what I've seen, it's also common to factor this expression as

$$ p(\theta_1, \theta_2) = p(\theta_1|\theta_2)p(\theta_2) $$

This requires no further independence assumptions, and it might be easy to specify the conditional distribution $p(\theta_1|\theta_2)$. Having said that, in many cases it can be completely justifiable to have independent priors on $\theta_1$ and $\theta_2$ so it really depends on the individual problem you're trying to solve.

$\endgroup$
3
$\begingroup$

As Maurits M mentions in another answer whether independence makes sense is really problem specific. The OP question asked:

"I am wondering why most set-ups assume that the priors above are independent. What happens if we do not have it?"

which is really 2 questions.

(1) Why most set-ups assume the priors are independent?

My guess (w/o being able to read minds) would be that multivariate distributions that can be written in closed form are few and far between. This is also the reason why MCMC techniques are so popular. It is much easier to write a product of marginal priors and may make the sampler easier to write down.

(2) What happens if we do not have it? This question could be interpreted as the impact of incorrectly specifying independence, or as to how to proceed if you know independence is not a reasonable assumption. I'll answer both.

If you've incorrectly assumed independence, then the degree of the impact will depend on how egregious this violation is w.r.t. the true underlying model. For example, naive Bayes assumes independence and often works well even if the independence is empirically not true. The reason is that often symmetries exist in the data generation mechanism which "cancel out" the independence violations. However, this statement is more of an empirical justification and I'm unaware of any group theoretic proof of this claim.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.