# How do Bayesians compare distributions?

So, I think that I have a decent grasp of the basics of frequentist probability and statistical analysis (and how badly it can be used). In a frequentist world, it makes sense to ask such a question as "is this distribution different from that distribution", because distributions are assumed to be real, objective and unchanging (for a given situation, at least), and so we can figure out how likely it is that one sample is drawn from a distribution shaped like another sample.

In the Bayesian world view, we only care about what we expect to see, given our past experiences (I'm still a bit vague on this part, but I understand the concept of Bayesian updating). If that is so, how can a Bayesian say "this set of data is different from that set of data"?

For the purposes of this question, I don't care about statistical significance, or similar, just how to quantify difference. I'm equally interested in parametric and non-parametric distributions.

• Could you clarify what you mean by "this set of data is different from that set of data?" As in, are you referring to the comparison of two or more groups, such as incomes of males vs. incomes of females? Or perhaps how does a Bayesian compare two samples of incomes without knowledge of gender? – ramhiser Jun 3 '13 at 16:49
• @JohnA.Ramey: What is the difference? Once it's all numbers, aren't "male" and "female" just labels for samples? – naught101 Jun 4 '13 at 0:13

Think your statement through as a Frequentist and make it more specific first. A Frequentist could not say that "data set A is different from data set B", without any further clarification.

First, you'd have to state what you mean by "different". Perhaps you mean "have different mean values". Then again, you might mean "have different variances". Or perhaps something else?

Then, you'd have to state what kind of test you would use, which depends on what you believe are valid assumptions about the data. Do you assume that the data sets are both normally-distributed about some means? Or do you believe that they are both Beta-distributed? Or something else?

Now can you see that the second decision is much like the priors in Bayesian statistics? It's not just "my past experience", but is rather what I believe, and what I believe my peers will believe, are reasonable assumptions about my data. (And Bayesians can use uniform priors, which pushes things towards Frequentist calculations.)

EDIT: In response to your comment: the next step is contained in the first decision I mentioned. If you want to decide whether the means of two groups are different, you would look at the distribution of the difference of the means of the two groups to see if this distribution does or does not contain zero, at some level of confidence. Exactly how close to zero you count as zero and exactly which portion of the (posterior) distribution you use are determined by you and the level of confidence you desire.

A discussion of these ideas can be found in a paper by Kruschke, who also wrote a very readable book Doing Bayesian Data Analysis, which covers an example on pages 307-309, "Are Different Groups Equal?". (Second edition: p. 468-472.) He also has a blog posting on the subject, with some Q&A.

FURTHER EDIT: Your description of the Bayesian process is also not quite correct. Bayesians only care about what the data tells us, in light of what we knew independent of the data. (As Kruschke points out, the prior does not necessarily occur before the data. That's what the phrase implies, but it's really just our knowledge excluding some of the data.) What we knew independently of a particular set of data may be vague or specific and may be based on consensus, a model of the underlying data generation process, or may just be the results of another (not necessarily prior) experiment.

• Yes, ok, frequentists assume a distribution, and that is subjective. But then they can just measure the parameters of each sample, with error, and say "ok, these are the parameters of the true population of each sample, and now what is the probability that the difference is just due to sampling error". My question is about the step after your answer - how do Bayesian infer differences between samples (let's assume the samples are from the same type of distribution, parametric or not). – naught101 Jun 7 '13 at 1:28
• @naught101: Please see my edit. – Wayne Jun 7 '13 at 19:02
• @Wayne the paper you linked is excellent. Thanks for sharing – Cam.Davidson.Pilon Jun 7 '13 at 21:15
• @naught101: I've updated the blog link. He's evidently kept older versions of the article and each links to a newer one, and the one I first linked to is three versions out of date. – Wayne Jun 8 '13 at 14:42
• This is quite a cool method, and it really makes clear how bayesian inference might work (by treating the distribution parameters as the source of uncertainty). Pity it's so computationally intensive. Also, the use of 95% CIs seems a bit too much like setting a significance level, but I can't see if there's a real way to get a reportable equivalent of a p-value (perhaps the sum of the probabilities of values more extreme than 0 from the mean, for the difference in means?). – naught101 Jun 13 '13 at 4:42

this paper might be of interest: http://arxiv.org/pdf/0906.4032v1.pdf

It gives a nice summary of some frequentist and Bayesian approaches to the two sample problem, and discusses both the parametric and nonparametric cases.

It might add something to the other answers to give a simple example. Say you have two data sets $\mathbf{x}$ and $\mathbf{y}$ where each $x_i$ and each $y_j$ is either a $0$ or a $1$. You assume an iid Bernoulli model in both cases, so each $x_i\sim Bern(p)$ and each $y_i\sim Bern(q)$. Your hypothesis testing scenario in both the frequentist and Bayesian settings can be:

$\mathcal{H}_0: \: \: p=q$

$\mathcal{H}_1: \: \: p,q$ not necessarily equal.

The likelihoods for the data in each case are:

Under $\mathcal{H}_0$ : $L_0(p) = f(\mathbf{x},\mathbf{y};p) = \prod_i p^i (1-p)^{1-i} \prod_j p^j(1-p)^{1-j}$

Under $\mathcal{H}_1$ : $L_1(p,q) = f(\mathbf{x},\mathbf{y};p,q) = \prod_i p^i (1-p)^{1-i} \prod_j q^j(1-q)^{1-j}$

(since under $\mathcal{H}_0 \:\: q=p$). A frequentist approach to the problem might be to do a Likelihood ratio test, whereby you calculate the statistic:

$W = -2\log\left\{ \frac{L_0(p_{max})}{L_1(p_{max},q_{max})}\right\},$

where $p_{max},q_{max}$ denote the maximum likelihood estimates for $p$ and $q$ under the relevant hypothesis (so $p_{max}$ in the numerator may not be the same as $p_{max}$ in the denominator). $W$ asymptotically follows a $\chi^2_1$ distribution (see e.g. Pawitan, 2001), so you would specify a significance level and reject/fail to reject $\mathcal{H}_0$ as appropriate.

Traditionally, in the Bayesian approach the test statistic would be the Bayes factor. First you would assume some relevant priors $p\sim \pi_0$ under $\mathcal{H}_0$ and $p,q\sim \pi_1$ under $\mathcal{H}_1$. The Bayes factor is the ratio of marginal likelihoods, given by:

$BF = \frac{ f(\mathbf{x},\mathbf{y}|\mathcal{H}_0) }{f(\mathbf{x},\mathbf{y}|\mathcal{H}_1)} = \frac{ \int_0^1 L_0(p)\pi_0(p)dp}{\int_0^1 \int_0^1 L_1(p,q)\pi_1(p,q)dpdq}$.

The Bayes factor can be combined with some prior beliefs on the probability of $\mathcal{H}_0$ or $\mathcal{H}_1$ being true, to give the probability of $\mathcal{H}_0$ versus $\mathcal{H}_1$ after seeing the data. If we assume apriori that each hypothesis is equally likely, so $p(\mathcal{H}_0)=p(\mathcal{H}_1) = 1/2$, then this gives:

$\frac{p(\mathcal{H}_0|\mathbf{x},\mathbf{y})}{p(\mathcal{H}_1|\mathbf{x},\mathbf{y})} = BF \times \frac{p(\mathcal{H}_0)}{p(\mathcal{H}_1)} = BF \times \frac{1/2}{1/2} = BF.$

Intuitively, if this ratio is $>1$, then the posterior probability of $\mathcal{H}_0$ is larger than $\mathcal{H}_1$, so you would say that $\mathcal{H}_0$ has a higher probability of being true under these assumptions for the prior and model.

One nice thing about the Bayes factor is how it automatically penalises more complex models (such as $\mathcal{H}_1$ here). A nice paper offering some more intuition is here: http://quasar.as.utexas.edu/papers/ockham.pdf.