0
$\begingroup$

Frequentism is very roughly the view that the probability of some attribute or event is identical to its actual relative frequency or hypothetical long-run, relative frequency.

My question: what is the relationship of significance tests in statistics to Frequentism?
(By "significance tests" I mean the general procedure of rejecting the null hypothesis H if the p-value of H and the observed test statistic is below the significance level alpha). Significance tests seem to me to be at the core of classical statics, which is usually strongly linked to Frequentism. So, are significance tests meaningless/unjustifiable if Frequentism is false and if not, why?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Hypothesis tests do not rely on a frequentist interpretation of probability for their validity. They do, however, avoid using Bayesian assumptions and that might be why some people associate these tests with frequentist approaches. $\endgroup$
    – whuber ♦
    Commented Nov 20, 2022 at 18:00
  • $\begingroup$ @whuber Usually I find myself in full agreement with your comments, However, in this case your point is not clear to me, I wonder if you can clarify. It is my understanding that the frequentist interpretation of p-values and hypothesis tests is what many find attractive. $\endgroup$
    – Graham Bornholt
    Commented Nov 22, 2022 at 4:17
  • $\begingroup$ David Cox in his "Principles of Statistical Inference" book (p.197) when writing positively about Frequentist analyses states: "The implications of data are examined using measuring techniques such as confidence limits and significance tests calibrated, as are other measuring instruments, indirectly by the hypothetical consequences of their repeated use". $\endgroup$
    – Graham Bornholt
    Commented Nov 22, 2022 at 4:17
  • $\begingroup$ @Graham That reads like a frequentist interpretation, but not a frequentist derivation or justification. The decision-theoretic justification (based on optimizing the risk function for a binary loss) does not require any specific interpretation of probability. (Bayesians object to classical hypothesis testing not because they interpret probability differently, but because they insist that the parameter must have a probability distribution.) From that perspective, the Cox quotation sounds only like a heuristic statement of a law of large numbers. $\endgroup$
    – whuber ♦
    Commented Nov 22, 2022 at 4:25
  • $\begingroup$ OK, you are drawing a distinction between frequentist interpretation and frequentist justification. However, if the desired frequentist interpretation was not present, Neyman-Pearson would not be interested in hypothesis tests, and Cox would not have been interested in significance tests. Sure, you can do Bayesian hypothesis testing but that is a different topic. Neyman-Pearson were optimizing across alternative tests that all had frequentist interpretability as I understand it. $\endgroup$
    – Graham Bornholt
    Commented Nov 22, 2022 at 4:45

1 Answer 1

Reset to default
1
$\begingroup$

When I say that a difference is "statistically significant at the 95% confidence level" (or that "p<.05") what that really means is...

"If, in reality, the difference in question were actually zero, and I had repeated this exact same analysis 100 different times, each time on a different random sample of the same size drawn from the same population, then only 5 of those 100 analyses would have (incorrectly) found a difference as large or larger than the one I actually observed."

So you can see that the concept of "taking repeated samples" is built right into the idea of statistical significance (and null hypothesis testing). In other words, saying something is statistically significant is really just making a claim about what would happen if you repeated the analysis you just did a large number of times.

It doesn't really make sense to worry about whether Frequentism is "true" or not. Frequentism and Bayesianism are two philosophical ways to think about this fuzzy concept called probability, but there is no way to ever prove that one of them is right or wrong. Of course, people often argue about which is more useful in particular contexts. But it is true that standard statistical tests, p values, and confidence intervals all presuppose a frequentist view of statistics.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Technically, your initial characterization is incorrect: it comes across as interpreting a probability as the actual number of occurrences. The concept of taking repeated samples is built into this mischaracterization, but it's not a part of the theory of hypothesis testing. $\endgroup$
    – whuber ♦
    Commented Nov 20, 2022 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.