Very interesting question, here's my take on it.

It's all about encoding information, then turn the Bayesian crank. It seems too good to be true - but both of these are harder than they seem.

I start with asking the question

What information is being used when we worry about multiple comparisons?

I can think of some - the first is "data dredging" - test "everything" until you get enough passes/fails (I would think almost every stats trained person would be exposed to this problem). You also have less sinister, but essentially the same "I have so many tests to run - surely all can't be correct".

After thinking about this, one thing I notice is that you don't tend to hear much about specific hypothesis or specific comparisons. It's all about the "collection" - this triggers my thinking towards exchangeability - the hypothesis being compared are "similar" to each other in some way. And how do you encode exchangeability into bayesian analysis? - hyper-priors, mixed models, random effects, etc!!!

But exchangeability only gets you part of the way there. Is everything exchangeable? Or do you have "sparsity" - such as only a few non-zero regression coefficients with a large pool of candidates. Mixed models and normally distributed random effects don't work here. They get "stuck" in between squashing noise and leaving signals untouched (eg in your example keep locationB and locationC "true" parameters equal, and set locationA "true" parameter arbitrarily large or small, and watch the standard linear mixed model fail.). But it can be fixed - eg with "spike and slab" priors or "horse shoe" priors.

So it's really more about describing what sort of hypothesis you are talking about and gettin as many known features reflected in the prior and likelihood. Andrew Gelman's approach is just a way to handle a broad class of multiple comparisons implicitly. Just like least squares and normal distributions tend to work well in most cases (but not all).

In terms of how it does this, you could think of a person reasoning as follows

• group A and group B might have the same mean
• I looked at the data, and the means are "close"
• Hence, to get a better estimate for both, I should pool the data, as my initial thought was they have the same mean.
• If they are not the same, the data provides evidence that they are "close", so pooling "a little bit" won't hurt me too badly if my hypothesis was wrong (ala all models are wrong, some are useful)

Note that all the above hinges on the initial premise "they might be the same". Take that away, and there is no justification for pooling. You can probably also see a "normalish distribution" way of thinking about the tests. "Zero is most likely", "if not zero, then close to zero is next most likely", "extreme values are unlikely". Consider this alternative:

• group A and group B means might be equal, but they could also be drastically different

Then the argument about pooling "a little bit" is a very bad idea. You are better off choosing total pooling or zero pooling. Much more like a Cauchy, spike&slab, type of situation (lots of mass around zero, and lots of mass for extreme values)

The whole multiple comparisons doesn't need to be dealt with, because the Bayesian approach is incorpating the information that leads us to worry into the prior and/or likelihood. In a sense it more a reminder to properly think about what information is available to you, and making sure you have included it in your analysis.

Very interesting question, here's my take on it.

It's all about encoding information, then turn the Bayesian crank. It seems too good to be true - but both of these are harder than they seem.

I start with asking the question

What information is being used when we worry about multiple comparisons?

I can think of some - the first is "data dredging" - test "everything" until you get enough passes/fails (I would think almost every stats trained person would be exposed to this problem). You also have less sinister, but essentially the same "I have so many tests to run - surely all can't be correct".

After thinking about this, one thing I notice is that you don't tend to hear much about specific hypothesis or specific comparisons. It's all about the "collection" - this triggers my thinking towards exchangeability - the hypothesis being compared are "similar" to each other in some way. And how do you encode exchangeability into bayesian analysis? - hyper-priors, mixed models, random effects, etc!!!

But exchangeability only gets you part of the way there. Is everything exchangeable? Or do you have "sparsity" - such as only a few non-zero regression coefficients with a large pool of candidates. Mixed models and normally distributed random effects don't work here. They get "stuck" in between squashing noise and leaving signals untouched (e.g. in your example keep locationB and locationC "true" parameters equal, and set locationA "true" parameter arbitrarily large or small, and watch the standard linear mixed model fail.). But it can be fixed - e.g. with "spike and slab" priors or "horse shoe" priors.

So it's really more about describing what sort of hypothesis you are talking about and getting as many known features reflected in the prior and likelihood. Andrew Gelman's approach is just a way to handle a broad class of multiple comparisons implicitly. Just like least squares and normal distributions tend to work well in most cases (but not all).

In terms of how it does this, you could think of a person reasoning as follows

• group A and group B might have the same mean
• I looked at the data, and the means are "close"
• Hence, to get a better estimate for both, I should pool the data, as my initial thought was they have the same mean.
• If they are not the same, the data provides evidence that they are "close", so pooling "a little bit" won't hurt me too badly if my hypothesis was wrong (a la all models are wrong, some are useful)

Note that all the above hinges on the initial premise "they might be the same". Take that away, and there is no justification for pooling. You can probably also see a "normalish distribution" way of thinking about the tests. "Zero is most likely", "if not zero, then close to zero is next most likely", "extreme values are unlikely". Consider this alternative:

• group A and group B means might be equal, but they could also be drastically different

Then the argument about pooling "a little bit" is a very bad idea. You are better off choosing total pooling or zero pooling. Much more like a Cauchy, spike&slab, type of situation (lots of mass around zero, and lots of mass for extreme values)

The whole multiple comparisons doesn't need to be dealt with, because the Bayesian approach is incorporating the information that leads us to worry into the prior and/or likelihood. In a sense it more a reminder to properly think about what information is available to you, and making sure you have included it in your analysis.