Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

While attending conferences, there has been a bit of a push by advocates of Bayesian statistics for assessing the results of experiments. It is vaunted as both more sensitive, appropriate, and selective towards genuine findings (fewer false positives) than frequentist statistics.

I have explored the topic somewhat, and I am left unconvinced so far of the benefits to using Bayesian statistics. Bayesian analyses were used to refute Daryl Bem's research supporting precognition, however, so I remain cautiously curious about how Bayesian analyses might benefit even my own research.

So I am curious about the following:

  • Power in a Bayesian analysis vs. a frequentist analysis
  • Susceptibility to Type 1 error in each type of analysis
  • The trade-off in complexity of the analysis (Bayesian seems more complicated) vs. the benefits gained. Traditional statistical analyses are straightforward, with well-established guidelines for drawing conclusions. The simplicity could be viewed as a benefit. Is that worth giving up?

Thanks for any insight!

share|improve this question

migrated from skeptics.stackexchange.com Mar 16 '11 at 7:27

This question came from our site for scientific skepticism.

1  
Bayesian statistics is traditional statistics - can you give a concrete example for what you mean be traditional statistics? –  Ophir Yoktan Mar 11 '11 at 19:54
1  
@OphirYoktan: He's talking about frequency probability versus Bayesian probability. It's even mentioned in the question's title. –  Borror0 Mar 11 '11 at 19:57
5  
I think this question should be moved over here: stats.stackexchange.com –  Mark L Mar 11 '11 at 23:29
2  
I asked a question on meta about whether this should be on-topic. –  Jason Plank Mar 13 '11 at 1:46
1  
I think this question can potentially have a "good" or "correct" answer. E.g. if someone could say "for every frequentist test with type 1 error $\alpha$ and type 2 error $\beta$, there exists a Bayesian test with type 1 error $\alpha$ and type 2 error $\beta - x$", this would be a good answer. Or something like "every frequentist test is equivalent to a Bayesian test with uninformative prior". I.e. this doesn't have to be a religious war between frequentists and bayesians. I'm only arguing because I don't understand how the replies relate to the specific questions in OP. –  SheldonCooper Mar 16 '11 at 23:56

3 Answers 3

A quick response to the bulleted content:

1) Power / Type 1 error in a Bayesian analysis vs. a frequentist analysis

Asking about Type 1 and power (i.e. one minus the probability of Type 2 error) implies that you can put your inference problem into a repeated sampling framework. Can you? If you can't then there isn't much choice but to move away from frequentist inference tools. If you can, and if the behavior of your estimator over many such samples is of relevance, and if you are not particularly interested in making probability statements about particular events, then I there's no strong reason to move.

The argument here is not that such situations never arise - certainly they do - but that they typically don't arise in the fields where the methods are applied.

2) The trade-off in complexity of the analysis (Bayesian seems more complicated) vs. the benefits gained.

It is important to ask where the complexity goes. In frequentist procedures the implementation may be very simple, e.g. minimize the sum of squares, but the principles may be arbitrarily complex, typically revolving around which estimator(s) to choose, how to find the right test(s), what to think when they disagree. For an example. see the still lively discussion, picked up in this forum, of different confidence intervals for a proportion!

In Bayesian procedures the implementation can be arbitrarily complex even in models that look like they 'ought' to be simple, usually because of difficult integrals but the principles are extremely simple. It rather depends where you'd like the messiness to be.

3) Traditional statistical analyses are straightforward, with well-established guidelines for drawing conclusions.

Personally I can no longer remember, but certainly my students never found these straightforward, mostly due to the principle proliferation described above. But the question is not really whether a procedure is straightforward, but whether is closer to being right given the structure of the problem.

Finally, I strongly disagree that there are "well-established guidelines for drawing conclusions" in either paradigm. And I think that's a good thing. Sure, "find p<.05" is a clear guideline, but for what model, with what corrections, etc.? And what do I do when my tests disagree? Scientific or engineering judgement is needed here, as it is elsewhere.

share|improve this answer
    
I'm not sure that asking about type 1/type 2 errors implies anything about a repeated sampling framework. It seems that even if my null hypothesis cannot be sampled repeatedly, it is still meaningful to ask about the probability of type 1 error. The probability in this case, of course, is not over all the possible hypotheses, but rather over all possible samples from my single hypothesis. –  SheldonCooper Mar 16 '11 at 15:24
    
It seems to me that the general argument is this: although making a type 1 (or 2) error can be defined for a 'one shot' inference (Type 1 vs 2 is just part of a typology of mistakes I can make) unless my making this mistake is embedded in repeated trials neither error type can have a frequentist probability. –  conjugateprior Mar 16 '11 at 15:53
    
What I'm saying is that making a type 1 (or 2) error is always embedded in repeated trials. Each trial is sampling a set of observations from the null hypothesis. So even if it is difficult to imagine sampling a different hypothesis, repeated trials are still there because it is easy to imagine sampling a different set of observations from that same hypothesis. –  SheldonCooper Mar 16 '11 at 15:59
    
You are, I think backing off to from N-P to Fisher when you ask whether it doesn't still make sense to think of the p value as simultaneously a measure of divergence from some null, as a way to define a type 1 error, and as something that doesn't require a sampling framework. I'm not sure how to have all these at once, but perhaps I'm missing something. –  conjugateprior Mar 16 '11 at 16:01
    
oops, we overlapped. In the terms of my original response to the question you're saying that you can imagine embedding in a repeated framework. OK, then Type 1 and perhaps Type 2 makes good sense. –  conjugateprior Mar 16 '11 at 16:04

Bayesian statistics can be derived from a few Logical principles. Try Searching "probability as extended logic" and you will find more in depth analysis of the fundamentals. But basically, Bayesian statistics rests on three basic "desiderata" or normative principles:

  1. The plausability of a proposition is to be represented by a single real number
  2. The plausability if a proposition is to have qualitative correspondance with "common sense". If given initial plausibility $p(A|C^{(0)})$, then change from $C^{(0)}\rightarrow C^{(1)}$ such that $p(A|C^{(1)})>p(A|C^{(0)})$ (A becomes more plausible) and also $p(B|A C^{(0)})=p(B|AC^{(1)})$ (given A, B remains just as plausible) then we must have $p(AB| C^{(0)})\leq p(AB|C^{(1)})$ (A and B must be at least as plausible) and $p(\overline{A}|C^{(1)})<p(\overline{A}|C^{(0)})$ (not A must become less plausible).
  3. The plausability of a proposition is to be calculated consistently. This means a) if a plausability can be reasoned in more than 1 way, all answers must be equal; b) In two problems where we are presented with the same information, we must assign the same plausabilities; and c) we must take account of all the information that is available. We must not add information that isn't there, and we must not ignore information which we do have.

These three desiderata (along with the rules of logic and set theory) uniquely determine the sum and product rules of probability theory. Thus, if you would like to reason according to the above three desiderata, they you must adopt a Bayesian approach. You do not have to adopt the "Bayesian Philosophy" but you must adopt the numerical results. The first three chapters of this book describe these in more detail, and provide the proof.

And last but not least, the "Bayesian machinery" is the most powerful data processing tool you have. This is mainly because of the desiderata 3c) using all the information you have (this also explains why Bayes can be more complicated than non-Bayes). It can be quite difficult to decide "what is relevant" using your intuition. Bayes theorem does this for you (and it does it without adding in arbitrary assumptions, also due to 3c).

EDIT: to address the question more directly (as suggested in the comment), suppose you have two hypothesis $H_0$ and $H_1$. You have a "false negative" loss $L_1$ (Reject $H_0$ when it is true: type 1 error) and "false positive" loss $L_2$ (Accept $H_0$ when it is false: type 2 error). probability theory says you should:

  1. Calculate $P(H_0|E_1,E_2,\dots)$, where $E_i$ is all the pieces of evidence related to the test: data, prior information, whatever you want the calculation to incorporate into the analysis
  2. Calculate $P(H_1|E_1,E_2,\dots)$
  3. Calculate the odds $O=\frac{P(H_0|E_1,E_2,\dots)}{P(H_1|E_1,E_2,\dots)}$
  4. Accept $H_0$ if $O > \frac{L_2}{L_1}$

Although you don't really need to introduce the losses. If you just look at the odds, you will get one of three results: i) definitely $H_0$, $O>>1$, ii) definitely $H_1$, $O<<1$, or iii) "inconclusive" $O\approx 1$.

Now if the calculation becomes "too hard", then you must either approximate the numbers, or ignore some information.

For a actual example with worked out numbers see my answer to this question

share|improve this answer
3  
I'm not sure how this answers the question. Frequentists of course disagree with desideratum 1 from this list, so the rest of the argument doesn't apply to them. It also doesn't answer any of the specific questions in the OP, such as "is Bayesian analysis more powerful or less error-prone than a frequentist analysis". –  SheldonCooper Mar 16 '11 at 15:18
    
@sheldoncooper - if a frequentist disagrees with desideratum 1, then on what basis can they construct a 95% confidence interval? They must require an additional number. –  probabilityislogic Mar 16 '11 at 22:32
    
@sheldoncooper - and further, sampling probabilities would have to be re-defined, because they too are only 1 number. A frequentist cannot reject desideratum 1 without rejecting their own theory –  probabilityislogic Mar 16 '11 at 23:12
1  
I'm not sure what additional number you are referring to. I'm also not sure what is the computation of $p(H_1|...)$ that you've introduced. The procedure in standard statistical tests is to compute $p(E_1, E_2, ... | H_0)$. If this probability is low, $H_0$ can be rejected; otherwise it cannot. –  SheldonCooper Mar 16 '11 at 23:21
1  
"they cannot reject desideratum 1 without rejecting their own theory" -- what do you mean by that? Frequentists have no notion of "plausibility". They have a notion of "frequency of occurrence in repeated trials". This frequency satisfies conditions similar to your three desiderata and thus happens to follow similar rules. Thus for anything for which the notion of frequency is defined, you can use laws of probability without any problem. –  SheldonCooper Mar 16 '11 at 23:25

I am not familiar with Bayesian Statistics myself but I do know that Skeptics Guide to the Universe Episode 294 has and interview with Eric-Jan Wagenmakers where they discuss Bayesian Statistics. Here is a link to the podcast: http://www.theskepticsguide.org/archive/podcastinfo.aspx?mid=1&pid=294

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.