I'm comparing two test methods, A and B, for asphalt density against the reference "true" density. I have paired measurements for each method and the reference. From a paired t-test of Method A vs. Reference, the p-value is 0.40. For Method B vs Reference, the p-value is 0.0007.

So, I reject the null hypothesis that Method B and the Reference are the same. I'm more than 95% confident that this is the case. But what about Method A? I fail to reject the null hypothesis....but that's not enough for me. I want to say that Method A values and the Reference values ARE the same. But can I do that? Do I need a bigger p-value? How large does the p-value need to be to conclude that Method A and the Reference are the same? Is "not different" equivalent to "being the same." Do I need a different null hypothesis?


  • $\begingroup$ It is just impossible to make such a statement. Consider the difference between "absence of evidence" and "evidence of absence". A high p-value refers to the former, what you want is the latter. $\endgroup$ – Maarten Buis Jan 26 '17 at 18:40

Technically, it is not possible to draw any conclusion about the "sameness" of the two groups from the p value. The p-value is the probability of observing the data giving that the null hypothesis is right. An high p-value means that assumes H0 is right simply means that given that hypothesis, it is very likely that you will be observing those data. Now let's pretend that you are testing the probability of observing the data given that the difference between your two groups is 10 (arbitrary number). You calculate your p-value it is still very high. What I'm trying to say is that you can have very high p-values of two different conclusions and they could be both plausible. Hence, you cannot draw any conclusion regarding the null hypothesis from the p-value.

What I would recommend in your case is to simply look at the raw differences. It provides more information about the actual resemblence between the groups. If you have sufficient sample size, you can compute the standard error of the mean (or their confidence intervals) and you can see if both group overlaps or compare the extreme scenarios (the biggest possible difference between the groups). If you have a small sample size or you think your data is not distributed normally, you can use bootstrapping to compute your confidence interval.

That would be one way to compare the two groups, but definitely not p-values.


I don't have enough points to comment right now, though I'm sure someone can come up with a more complete answer.

The simple answer is no you can't claim that reference and Method A are the same. There is no p-value that will allow you to make that claim. Your alternative hypothesis was that the $\mu_A \neq \mu_{Ref}$, and with that, all you can say is that there is not statistically significant evidence to suggest that the means are different.

I'm not aware of a test wherein the alternative hypothesis is that the means are equal, but that's not to say one doesn't exist. You should report confidence intervals to show what plausible values for $\mu_A$ are -- it still won't give you what you want but will give you a better idea of the distribution of $\mu_A$.

  • $\begingroup$ I can appreciate that I can't prove populations are equivalent. I guess I want to know if there are methods to interpret high p-values. Can I interpret a p-value of 0.9 differently than a p-value of 0.3? $\endgroup$ – Bryan Jan 26 '17 at 19:06
  • $\begingroup$ I could also do linear regression (y=mx+b) between the Methods and References. A strong relationship should have m~1 and b~0, with a high R^2. Though this is more a discuss and not a statistical test. $\endgroup$ – Bryan Jan 26 '17 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.