I'm comparing two test methods, A and B, for asphalt density against the reference "true" density. I have paired measurements for each method and the reference. From a paired t-test of Method A vs. Reference, the p-value is 0.40. For Method B vs Reference, the p-value is 0.0007.
So, I reject the null hypothesis that Method B and the Reference are the same. I'm more than 95% confident that this is the case. But what about Method A? I fail to reject the null hypothesis....but that's not enough for me. I want to say that Method A values and the Reference values ARE the same. But can I do that? Do I need a bigger p-value? How large does the p-value need to be to conclude that Method A and the Reference are the same? Is "not different" equivalent to "being the same." Do I need a different null hypothesis?