11
$\begingroup$

It seems that there is an ongoing debate within the Bayesian community about whether we should be doing Bayesian parameter estimation or Bayesian hypothesis testing. I'm interested in soliciting opinions about this. What are the relative strengths and weaknesses of these approaches? In which contexts is one more appropriate than the other? Should we be doing both parameter estimation and hypothesis testing, or just one?

$\endgroup$
  • 1
    $\begingroup$ Parameter estimation and hypothesis testing are different things. I never heard of such debate and I don't know what it would be about? It's like you asked if it is better to eat a dinner, or go for a swim instead. $\endgroup$ – Tim Nov 17 '16 at 12:23
  • 1
    $\begingroup$ No, he does not make such argument. He shows how to estimate Bayesian t-test. If you need to estimate parameter, then you need to estimate parameter, if you need to test a hypothesis, then you need to test a hypothesis, you do not use them interchangeably. $\endgroup$ – Tim Nov 17 '16 at 12:34
  • 1
    $\begingroup$ The paper is called "Bayesian estimation supersedes the t test". "Supersede" means "in the place of". Ergo, use Bayesian estimation in the place of (instead of) a t test. $\endgroup$ – sammosummo Nov 17 '16 at 13:32
  • 2
    $\begingroup$ @sammosummo Are you thinking of something like this Kruschke paper? $\endgroup$ – Ian_Fin Nov 17 '16 at 14:09
  • 1
    $\begingroup$ @Ian_Fin Yes that's exactly what I was thinking about, thank you. I should've checked Kruschke's other publications! I know that he, like Andrew Gelman, is strongly pro estimation and thought I might get more balanced arguments from Cross Validated. $\endgroup$ – sammosummo Nov 17 '16 at 18:04
9
$\begingroup$

In my understanding, the problem is not about opposing parameter estimation or hypothesis testing that indeed answers different formal questions but more about how science should work and more specifically what statistical paradigm should we use to answer a given practical question.

Most of the time, hypothesis testing is used : you want to test a new drug, you test $H_O:$ "it effect is similar to a placebo". However, you can also formalize it as: "what is the range of probable effect of the drug ?" which leads you to inference and particularly interval (hpd) estimation. This transposes the original question in a different but maybe more interpretation prone manner. Several notorious statisticians advocate for "such a" solution (e.g. Gelman see http://andrewgelman.com/2011/04/02/so-called_bayes/ or http://andrewgelman.com/2014/09/05/confirmationist-falsificationist-paradigms-science/).

More elaborated aspects of Bayesian inference for such testing purpose includes:

$\endgroup$
  • 1
    $\begingroup$ (+1) Thanks for connecting to our paper! I was wondering whether to mention this aspect... $\endgroup$ – Xi'an Nov 17 '16 at 14:02
  • 1
    $\begingroup$ +1 but it might be good to add some links to people (unlike Gelman) advocating against Bayesian estimation and in favour of Bayesian hypothesis testing. I have some links in my answer to stats.stackexchange.com/questions/200500. EJ Wagenmakers is I think one person who is very much in the Bayesian testing camp. See Why hypothesis tests are essential for psychological science: A comment on Cumming and possibly his other papers. $\endgroup$ – amoeba Jan 10 '17 at 15:37
  • $\begingroup$ I found your answer to the previous question before I asked this one. It is an excellent answer (and excellent question) and both of them completely supersede mine. $\endgroup$ – sammosummo Jan 10 '17 at 17:53
  • $\begingroup$ I think peuhp meant "famous statisticians" not "notorious statisticians". But maybe not! :-) Anyway, if people follow peuhp's link to the posterior predictive check advocated by Gelman and Shalizi, then people should also consider the comments on that article, one of which is here: indiana.edu/~kruschke/articles/Kruschke2013BJMSP.pdf $\endgroup$ – John K. Kruschke Jan 11 '17 at 0:25
8
$\begingroup$

In complement to peuhp's excellent answer, I want to add that the only debate I am aware of is whether or not hypothesis testing should be part of the Bayesian paradigm. This debate has been going on for decades and is not new. The arguments against producing a definitive answer to the question "is the parameter $\theta$ within a subset $\Theta_0$ of the parameter space?" or to the question "is model $\mathscr{M}_1$ the model behind the given data?" are many and, in my opinion, compelling enough to be considered. For instance, in a recent paper, as pointed out by peuhp, we argue that model choice and hypothesis testing can be conducted via an embedding mixture model that can be estimated, the relevance of each model or hypothesis for the data at hand being translated by the posterior distribution on the weights of the mixture, which can be seen as an "estimation".

The traditional Bayesian procedure for testing hypotheses is to return a definitive answer based on the posterior probability of the said hypothesis or model. This is formally validated by a decision-theory argument using Neyman-Pearson's $0-1$ loss function, which penalises all wrong decisions with the same loss. Given the complexity of the model choice and hypothesis testing settings, I find this loss function much too rudimentary to be compelling.

After reading Kruschke's paper, it seems to me that he opposes an approach based on HPD regions to the use of a Bayes factor, which sounds like the Bayesian counterpart of the frequentist opposition between Neymann-Pearson testing procedures and inverting confidence intervals.

$\endgroup$
3
$\begingroup$

As previous respondents have said, (Bayesian) hypothesis testing and (Bayesian) continuous-parameter estimation provide different information in response to different questions. There may be some occasions in which the researcher really needs an answer to a test of a null hypothesis. In this case, a carefully conducted Bayesian hypothesis test (using meaningfully informed, non-default priors) can be very useful. But all too often null-hypothesis tests are "mindless rituals" (Gigerenzer et al.) and make it easy for the analyst to lapse into fallacious "black and white" thinking about presence or absence of effects. A preprint at OSF provides an extended discussion of frequentist and Bayesian approaches to hypothesis testing and estimation with uncertainty, organized around this table: enter image description here You can find the preprint here: https://osf.io/dktc5/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.