0
$\begingroup$

I was planning on running a statistical test for hypothesis testing, but was confused if statistical power is important once a test is run. enter image description here

Looking at this confusion matrix, one would ideally set alpha to be a small number and beta to be close to 1 to avoid the errors. Test power (1-B) has been suggested in other online posts to only be only useful before a test is created/run by some, and useful only in determining the test conditions/number of samples required to get statistically significant results..

My confusion is if it makes sense to not accept a rejected null hypothesis as statistically significant evidence for a null hypothesis being false due to the test having insufficient power. I've heard two interpretations:

  1. It doesn't make sense to not accept a rejected null hypothesis as evidence because the null hypothesis is 100% true or false (conditioning on the columns), and once we complete the test, the test 100% either rejected or accepting the null hypothesis (conditioning on rows), so which "box" you are in is already known once the test completes. The power is only useful in increasing the likelyhood you reject a null hypothesis correctly while testing.
  2. It makes sense to not accept a rejected null hypothesis because we condition on only the outcome (rejected/not rejected) observed in the test (rows). For a rejected null hypothesis, it could either be because of a Type 1 error, or be the correct decision. The null hypothesis is not known to be 100% true or false (columns) and the conditional probability is the prior probability of the null hypothesis actually being true/false multiplied by alpha and 1-B respectively. Since power (1-B) is part of the probability calculation, it is important to determine if a rejected null hypothesis really means that the null hypothesis is false.

From what I understand, 1 and 2 are the Frequentist and Bayesian interpretations of the test respectively.

Which of these two interpretations (or neither) is correct?

EDIT: It seems my question has been answered here and here. The Bayesian interpretation is correct.

$\endgroup$
3
  • 2
    $\begingroup$ Ok, if I understand correctly, your question is "If I end up rejecting the null hypothesis for a underpowered test, should I believe the null is false". Is that correct? $\endgroup$ Commented Nov 20, 2023 at 0:57
  • $\begingroup$ Your description of point 1 as a frequentist is really a caricature of the frequentist approach to testing. Testing a null hypothesis can be regarded as the frequentist procedure to answer the question: Are the data reasonably consistent with the null hypothesis? If not consistent (p-value < alpha) then the null is rejected (provisionally of course). An alternative hypothesis may or may not be specified. Note that however close or far the alternative hypothesis is, the p-value is unchanged. Thus, the p-value can be regarded as a measure of the absolute plausibility of the null hypothesis. $\endgroup$ Commented Nov 21, 2023 at 2:59
  • $\begingroup$ Testing is not about the relative plausibility of hypotheses. If the alternative hypothesis is composite and you wish to do a power analysis to help determine the sample size for the test, then a parameter value is chosen just for that purpose but it need have no special importance. $\endgroup$ Commented Nov 21, 2023 at 3:01

0