I was wondering if anyone had recommendations for papers or resources on bayesian analysis of frequentist hypothesis testing and use of p-values?

Brad Efron has this quote

One definition is says that a frequentist is a a Bayesian trying to do well, or at least not too badly, against any possible prior distribution

It's clear to me how this makes sense in the context of decision theory, but often I read things similar to:

Cosma Shalizi: If you find a small p-value, yay; you've got enough data, with precise enough measurement, to detect the effect you're looking for, or you're really unlucky

What does this really mean (in a precise mathematical sense)? "Enough" data for what? For low enough errors if we take the estimate to be true? And what does "precise enough...to detect" mean? To me that only seems to make sense in the context of a bayesian posterior distribution around the estimate.

In the simplest case I can think of, consider two hypotheses, $H_0$ and $H_A$ with $\alpha = 0.05$ and $1-\beta = 0.8$. When we "reject the null" , the probability that $H_0$ is true is: $\frac{P(H_0)*\alpha}{P(H_A)*(1-\beta) + P(H_0) * \alpha}$ where $P$ is a prior distribution. Using the fact that $P(H_0) = 1-P(H_A)$, we can see how the probability that $H_0$ is true changes given various parameters of $\alpha$ and $\beta$.

The plot below, for example, shows the problem of low power testing (see the replication crisis). However, it's not as clear to me how to interpret use of p-values in the more common, complex practice (in which the parameter is continuous, p-values inform data decisions or decision to look for more data, etc...)

enter image description here

  • 1
    $\begingroup$ Not a lengthy answer here, but the error he is likely referring to is actually related to a p-value. For example, in regression, the estimate divided by the standard error is the t-statistic. When this is 2.0 or above (actually 1.96) the you can say the effect is "significant." Of course, there are also issues of degrees of freedom to get the p-value. That said, a "significant" finding can very generally be thought of as the estimate being twice the SE. So, what if the the estimate has slightly more error due to measurement, sample size, or whatever ? $\endgroup$ – D_Williams Oct 2 '16 at 16:23
  • $\begingroup$ Sure I know the definition of statistically significant (data unlikely under the null hypothesis) but people seem to use the word so often as if it has some additional inferential content (aside from use in controlling Type I and Type II errors). To me, the word "significant" would imply that we can place reasonable "trust" in an estimate, but the normal sense of "statistically significant" doesn't tell us that by itself, we also need a prior. Is there some implicit prior people are using? $\endgroup$ – convolutedstatistic Oct 2 '16 at 19:27
  • 1
    $\begingroup$ A nice discussion of p-values and how they depend on your sample here $\endgroup$ – Pascal Oct 4 '16 at 10:16
  • $\begingroup$ Thank you Pascal, that was a nice read! The section on why p-values often "work" is excellent and also shows why p-values can be such an annoying issue: even though technically they should not be directly used to make statements about the probability of a hypothesis, a lot of the time it works out anyways. $\endgroup$ – convolutedstatistic Oct 5 '16 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.