31
$\begingroup$

I've been researching the Bayesian take on classical statistics lately. After reading about the Bayes factor I've been left wondering if power analysis is a necessity in this view of statistics. My main reason for wondering this is the Bayes factor really just appears to be a likelihood ratio. Once it's like 25:1 it seems like I can call it a night.

Am I far off? Any other reading I can do to learn more? Currently reading this book: Introduction to Bayesian Statistics, by W.M. Bolstad (Wiley-Interscience; 2nd ed., 2007).

Improve this question
$\endgroup$
2

4 Answers 4

Reset to default
13
$\begingroup$

Power is about the long run probability of p < 0.05 (alpha) in studies when the effect does exist in the population. In Bayes the evidence from study A feeds into priors for study B, etc. on down the line. Therefore, power as is defined in frequentist statistics doesn't really exist.

That said, it doesn't mean a justification for an N in a study shouldn't be provided. Even without Bayes power analysis is often a poor justification.

Improve this answer
$\endgroup$
5
  • 22
    $\begingroup$ A less limited view of power sees it as an expression of the risk curve for a 0-1 loss function. A Bayesian analysis integrates that risk over the prior probability. However, good Bayesian analyses consider the sensitivity of their results to the choice of prior distribution. That would seem to place us right back into the domain of power analysis. Although it might not go by that name and would be computed differently, the purpose would be the same: namely, determining how large a sample to obtain in order to be reasonably sure of meeting the study objectives. $\endgroup$
    – whuber ♦
    Apr 9, 2016 at 19:01
  • 3
    $\begingroup$ That's a very good point Whuber. However, that's not the only reason for power calculations and many a Bayesian would argue that it's unnecessary because you don't need to determine N beforehand (a mistake). $\endgroup$
    – John
    Apr 9, 2016 at 19:22
  • 10
    $\begingroup$ I'm puzzled by this. How could one possibly go about designing an experiment, obtaining resources, and planning it without having some kind of idea of what $N$ should be? $\endgroup$
    – whuber ♦
    Apr 9, 2016 at 19:25
  • 2
    $\begingroup$ I didn't come up with the idea, there are a number of papers where Bayesians argue that you can add subjects until you have strong enough evidence to make a decision as opposed to frequentist testing where such test and add procedures don't work. I could look up a reference I suppose. Most notably this comes up in proposing how to analyze data in clinical trials. $\endgroup$
    – John
    Apr 9, 2016 at 19:27
  • 11
    $\begingroup$ It's clear that when one can proceed in such a sequential manner, there can be excellent reasons to do so. But even that doesn't imply that estimating $N$ is "unnecessary"! I can even believe in the existence of people who might make such an argument, but I would be obliged to think of them as other-worldly creatures who have no interest in actual experimentation or data collection! $\endgroup$
    – whuber ♦
    Apr 9, 2016 at 19:30
22
$\begingroup$

You can perform hypothesis tests with Bayesian statistics. For example, you could conclude an effect is greater than zero if more than 95% of the posterior density is greater than zero. Or alternative, you could employ some form of binary decision based on Bayes factors.

Once you establish such a decision making system, it is possible to assess statistical power assuming a given data generating process and sample size. You could readily assess this in a given context using simulation.

That said, a Bayesian approach often focuses more on the credibility interval than the point estimate, and degree of belief rather than a binary decision. Using this more continuous approach to inference, you could instead assess other effects on inference of your design. In particular, you might want to assess the expected size of your credibility interval for a given data generating process and sample size.

Improve this answer
$\endgroup$
1
  • $\begingroup$ "…degree of belief rather than a binary decision." Do Bayesians not need to make binary decisions? $\endgroup$
    – Alexis
    Apr 14 at 5:56
5
$\begingroup$

This issue leads to a lot of misunderstandings because people use Bayesian stats to ask frequentist questions. For example, people want to determine if variant B is better than variant A. They can answer this question with Bayesian stats by determining if the 95% highest density interval of the difference between those two posterior distributions (B-A) is greater than 0 or a region of practical significance around 0. If you use bayesian stats to answer frequentist questions, however, you can still make frequentist errors: type I (false positives; opps - B isn't actually better) and type II (miss; fail to realize that B is truly better).

The point of a power analysis is reduce type II errors (e.g. have at least an 80% chance of finding an effect if it exists). A power analysis should also be used when using Bayesian stats to ask frequentist questions like the one above.

If you don't use a power analysis, and then you repeatedly peek at your data while collecting it and then stop only once you find a significant difference, then you are going to make more type I (false alarms) errors than you may expect - same as if you had been using frequentist statistics.

check out:

https://doingbayesiandataanalysis.blogspot.com/2013/11/optional-stopping-in-data-collection-p.html

http://varianceexplained.org/r/bayesian-ab-testing/

Of note - Some Bayesian approaches can reduce, but not eliminate, the probability of making a type I error (e.g., an appropriate informative prior).

Improve this answer
$\endgroup$
1
$\begingroup$

The need for a power analysis in a clinical trial for example is to be able to calculate/estimate how many participants to recruit to have a chance of finding a treatment effect (of a given minimum size) if it exists. It isn't feasible to recruit an endless number of patients, first because of time constraints and second because of cost constraints.

So, imagine we are taking a Bayesian approach to said clinical trial. Although flat priors are in theory possible, sensitivity to the prior is advisable anyway since, unfortunately, more than one flat prior is available (which is odd I'm now thinking, as really there should only be one way of expressing utter uncertainty).

So, imagine that, further, we do a sensitivity analysis (the model and not just the prior would also be under scrutiny here). This involves simulating from a plausible model for 'the truth'. In classical/Frequentist statistics, there are four candidates for 'the truth' here: H0, mu=0; H1, mu!=0 where either are observed with error (as in our real world), or without error (as in the unobservable real world). In Bayesian statistics, there are two candidates for 'the truth' here: mu is a random variable (as in the unobservable real world); mu is a random variable (as in our observable real world, from an uncertain individual's point of view).

So really it depends who you're trying to convince A) by the trial and B) by the sensitivity analysis. If it's not the same person, that would be quite strange.

What is actually in question is a consensus on what truth is and on what substantiates tangible evidence. The shared ground is that signature probability distributions are observable in our real observable world that do in some way evidently have some underlying mathematical truth that just happens to be so by chance, or is by design. I will stop there as this isn't an Arts page, but rather a Science page, or that's my understanding.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.