3
$\begingroup$

Say we are interested in assessing the effectiveness of two different methods on improving some measure $x$.

We randomly assign a group of 40 individuals to method 1 and a group of 40 individuals to method 2. We take a measurement pre intervention and post intervention. Following the intervention, we conduct a 2 samples t-test and get a p-value of 0.078. Is it correct to claim that both approaches are equally effective? I have seen this in a paper and don't agree that this is the correct interpretation (at least not when based on the p-value alone).

My perspective

I know NHST effectively gives us the following: $p(\text{data} \mid H_0)$. With that in mind, I do not think that claiming equivalence solely on the basis of an insiginificant p-value is strictly correct.

Why? Well, first of all, the hypothesis test answers the question "are the differences observed large enough to be considered surprising, assuming the null is true". It does not answer the question "are the differences exactly 0". In other words, we can observe differences >0 OR <0 and still obtain an insignificant p-value. Clearly this does not mean the difference in the two methods is =0 OR that the methods are equivalent. Now let's just say that in our example, that group 1 experiences a larger change in measure $x$ than group 2 (and we see this reflected by effect size or in the raw change scores), and yet we observe an insignificant p-value of 0.078. I can understand why some make the argument that the two methods can more or less be considered "equivalent" (for lack of a better term), because the true difference may well be 0 and observed difference is likely due to sampling variation, measurement error, or some combination of both. BUT, it is also possible that this is a type II error: If we had a larger sample size, would this difference be significant? Yes, at some-point it would.

So, for me, claims of equivalence should be based on either (a) inferiority/equivalence hypothesis testing or (b) as a minimum, interpretation of effect sizes/ magnitude of change scores, and NOT just on the basis of p-values. Yes, if the effect is large enough it will still be significant with smaller samples, but my point pertains more to using the p-value alone to make this claim.

I just find the claim of equivalence quite bold and strong when it is based solely on the magnitude of a p-value. I do, however, accept that this claim has more weight if the researcher has purposefully conducted an equivalence hypothesis test. What does everyone else think? Am I way off? Is this a fair comment or not?

$\endgroup$
5
  • 1
    $\begingroup$ You’re right that it’s wrong to claim equivalence. The statistical test shows an absence of evidence for a difference between methods. That’s different from evidence of an absence of difference. $\endgroup$ Mar 28, 2021 at 17:45
  • $\begingroup$ @AryaMcCarthy Thanks for the rapid response. You have said that far more succinctly than I could, but glad I am not ranting for no reason here! I think the issue is that the null hypothesis for the 2 samples t-test is often phrased online as "the two groups are equal". If you don't understand the actual interpretation of the p-value this is misleading. The second thing is that we also don't know whether our insignificant p-value is because there really is no true difference and our results are expected under normal variability, or we have made a type II error. Frequentist life! $\endgroup$ Mar 28, 2021 at 17:50
  • $\begingroup$ You will also find many similar incorrect claims that "the null hypothesis is true" reported in the literature when people perform specification tests. Typical incorrect statements: "The model is homoscedastic by the *** test"; "The model is linear by the *** test"; "The residuals are independent by the *** test"; "The effects are fixed, not random, by the *** test"; etc etc. $\endgroup$ Mar 30, 2021 at 11:15
  • $\begingroup$ Relevant. $\endgroup$
    – Alexis
    Nov 2, 2021 at 15:55
  • $\begingroup$ Also, @BigBendRegion I quite agree. I try to teach my students that test are always tests for the alternative… that's always what the conclusion is about (evidence for, or absence of evidence for $\text{H}_{\text{A}}$). $\endgroup$
    – Alexis
    Nov 2, 2021 at 16:07

1 Answer 1

2
$\begingroup$

As "equivalence testing" has a well-established meaning in statistics, you are correct that failure to reject a null hypothesis should not lead to a claim that the treatments are "equivalent" or "equally effective." Such a claim is properly based on a pre-chosen maximum difference that can be taken to be "equivalent" in practice, based on knowledge of the subject matter. As Walker and Nowacki ("Understanding Equivalence and Noninferiority Testing," J. General Internal Medicine 26: 192–196, 2011) emphasized:

The determination of the equivalence margin, $\delta$, is the most critical step in equivalence/noninferiority testing... It must be stressed that the value of the equivalence margin should be determined before the data is recorded. This is essential to maintain the type I error at the desired level.

You can't simply use the results of a standard t-test to establish equivalence post-hoc. They go on to start the next section of their review:

NO DIFFERENCE DOES NOT IMPLY EQUIVALENCE

Using a traditional comparative test to establish equivalence/noninferiority leads frequently to incorrect conclusions. The reason is two-fold. First, the burden of the proof is on the wrong hypothesis, i.e., that of a difference. In this setting, a significant result establishes a difference, whereas a nonsignificant result implies only that equivalency (or equality) cannot be ruled out. Consequently, the risk of incorrectly concluding equivalence can be very high. The other reason is that the margin of equivalence is not considered, and thus the concept of equivalence is not well defined.

The literature is clearly on your side here.

$\endgroup$
1
  • $\begingroup$ Thank you for such a comprehensive response and for directing me to these resources. I think they will prove really useful for my own learning. I think the difficulty in making such a claim is evident even from an error perspective. There are two circumstances when you accept the null-hypothesis. Either (a) You are correct; the true difference is ~0 and the observed difference is consistent with expected sampling variability/measurement error. Or (b), you have made a Type II error. You cannot know which, so why make such a bold claim? Thanks again for your response here! $\endgroup$ Mar 28, 2021 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.