# Does statistically insignificant difference of means imply equality of means?

Suppose I have sets A and B of normally distributed data so that:

 A: mean=250, SD=200, N=25
B: mean=248, SD=200, N=20


Clearly, there is no statistically significant difference between the means (p=0.9736). But does this mean that the means are equal or that we have no evidence to suggest otherwise? Considering the huge standard deviation it seems intuitively unlikely that the means are equal. But this is what the null hypothesis says and based on the the t-test we didn't reject that.

Having such noisy data, how can one quantify how equal the means are?

Update This question arose from a discussion between myself and a colleague who was testing the effect of two drug treatments on biological samples. As the test revealed an insignificant difference between treatments, my colleague assumed this allows him to claim that the treatments have the same effect. With the example above I tried to show that this is not necessarily the case. But I didn't know the proper terminology to make things more quantitative. Hence this question.

• I've added the equivalence tag, and I would suggest you check out those other questions as well. For equivalence testing you typically need to specify the bounds for the hypothesis test where you would consider them to be equal. This may be good enough to "quantify how equal the means are" - but that statement is a bit vague. – Andy W May 18 '14 at 18:38
• Thanks for the tip! I agree, my question is quite vague, but that's because I'm not sure how to formulate it properly or what terms to use in this case. At what significance level are the means equal? But the t-test already tells us the level at which the difference is not significant... I guess I needed to know what is the terminology that applies to this type of question. Equivalence testing sounds promising! – numentar May 18 '14 at 18:51
• I take "how equal" to be a bit of an odd way to put it. We can estimate the difference between the means, $\mu_A - \mu_B$, and estimate the uncertainty around that difference, but how does one thing be more equal than another. Typically we test the hypothesis $\mu_A - \mu_B = 0$, but in non-inferiority or equivalence testing we flip the hypothesis to be something like $|\mu_A - \mu_B| < 2$, e.g. the absolute value of the difference of the means is less than 2 (for one possible example). The number you choose should be based on other criteria - not based on the observed data. – Andy W May 18 '14 at 19:03
• This question asks the same in terms of confidence intervals. Those are generally more instructive than $p$-values, because they live on the scale you measured, so you should prefer the confidence intervals anyway. – Horst Grünbusch May 18 '14 at 20:02
• "But does this mean that the means are equal or that we have no evidence to suggest otherwise?" -- the second. In no way does it imply the first. Indeed, in a very practical sense it can almost never be the case (an exception might be where population parameters may be discrete for some reason). – Glen_b May 19 '14 at 0:54