Say we are interested in assessing the effectiveness of two different methods on improving some measure $x$.
We randomly assign a group of 40 individuals to method 1 and a group of 40 individuals to method 2. We take a measurement pre intervention and post intervention. Following the intervention, we conduct a 2 samples t-test and get a p-value of 0.078. Is it correct to claim that both approaches are equally effective? I have seen this in a paper and don't agree that this is the correct interpretation (at least not when based on the p-value alone).
My perspective
I know NHST effectively gives us the following: $p(\text{data} \mid H_0)$. With that in mind, I do not think that claiming equivalence solely on the basis of an insiginificant p-value is strictly correct.
Why? Well, first of all, the hypothesis test answers the question "are the differences observed large enough to be considered surprising, assuming the null is true". It does not answer the question "are the differences exactly 0". In other words, we can observe differences >0 OR <0 and still obtain an insignificant p-value. Clearly this does not mean the difference in the two methods is =0 OR that the methods are equivalent. Now let's just say that in our example, that group 1 experiences a larger change in measure $x$ than group 2 (and we see this reflected by effect size or in the raw change scores), and yet we observe an insignificant p-value of 0.078. I can understand why some make the argument that the two methods can more or less be considered "equivalent" (for lack of a better term), because the true difference may well be 0 and observed difference is likely due to sampling variation, measurement error, or some combination of both. BUT, it is also possible that this is a type II error: If we had a larger sample size, would this difference be significant? Yes, at some-point it would.
So, for me, claims of equivalence should be based on either (a) inferiority/equivalence hypothesis testing or (b) as a minimum, interpretation of effect sizes/ magnitude of change scores, and NOT just on the basis of p-values. Yes, if the effect is large enough it will still be significant with smaller samples, but my point pertains more to using the p-value alone to make this claim.
I just find the claim of equivalence quite bold and strong when it is based solely on the magnitude of a p-value. I do, however, accept that this claim has more weight if the researcher has purposefully conducted an equivalence hypothesis test. What does everyone else think? Am I way off? Is this a fair comment or not?
