5
$\begingroup$

Suppose I have sets A and B of normally distributed data so that:

 A: mean=250, SD=200, N=25 
 B: mean=248, SD=200, N=20

Clearly, there is no statistically significant difference between the means (p=0.9736). But does this mean that the means are equal or that we have no evidence to suggest otherwise? Considering the huge standard deviation it seems intuitively unlikely that the means are equal. But this is what the null hypothesis says and based on the the t-test we didn't reject that.

Having such noisy data, how can one quantify how equal the means are?

Update This question arose from a discussion between myself and a colleague who was testing the effect of two drug treatments on biological samples. As the test revealed an insignificant difference between treatments, my colleague assumed this allows him to claim that the treatments have the same effect. With the example above I tried to show that this is not necessarily the case. But I didn't know the proper terminology to make things more quantitative. Hence this question.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ I've added the equivalence tag, and I would suggest you check out those other questions as well. For equivalence testing you typically need to specify the bounds for the hypothesis test where you would consider them to be equal. This may be good enough to "quantify how equal the means are" - but that statement is a bit vague. $\endgroup$
    – Andy W
    May 18, 2014 at 18:38
  • $\begingroup$ Thanks for the tip! I agree, my question is quite vague, but that's because I'm not sure how to formulate it properly or what terms to use in this case. At what significance level are the means equal? But the t-test already tells us the level at which the difference is not significant... I guess I needed to know what is the terminology that applies to this type of question. Equivalence testing sounds promising! $\endgroup$
    – numentar
    May 18, 2014 at 18:51
  • 1
    $\begingroup$ I take "how equal" to be a bit of an odd way to put it. We can estimate the difference between the means, $\mu_A - \mu_B$, and estimate the uncertainty around that difference, but how does one thing be more equal than another. Typically we test the hypothesis $\mu_A - \mu_B = 0$, but in non-inferiority or equivalence testing we flip the hypothesis to be something like $|\mu_A - \mu_B| < 2$, e.g. the absolute value of the difference of the means is less than 2 (for one possible example). The number you choose should be based on other criteria - not based on the observed data. $\endgroup$
    – Andy W
    May 18, 2014 at 19:03
  • $\begingroup$ This question asks the same in terms of confidence intervals. Those are generally more instructive than $p$-values, because they live on the scale you measured, so you should prefer the confidence intervals anyway. $\endgroup$
    – Horst Grünbusch
    May 18, 2014 at 20:02
  • 1
    $\begingroup$ "But does this mean that the means are equal or that we have no evidence to suggest otherwise?" -- the second. In no way does it imply the first. Indeed, in a very practical sense it can almost never be the case (an exception might be where population parameters may be discrete for some reason). $\endgroup$
    – Glen_b
    May 19, 2014 at 0:54

1 Answer 1

Reset to default
6
$\begingroup$

Formally, no. Your conclusion is only that you "cannot reject" the null that they are equal. They might still be different! and more data could, maybe, show that.

To say more than such formalities, we need to know more about the context. In your applied setting, with your intended use of the results, how large must a difference be to have any practical significance? To help getting into that mindset, it is better for you to calculate a confidence interval (say, 95%) on the difference of the means. If all the values within that interval are small enough for you (and your readers) to conclude "such a difference is to small to have any real import", then, and first then, can you conclude that the means are practically equal.

Empirical measurements can never decide mathematical equality, that is, identity!

As another commenter says, if you really want to prove equality (alternatively, "non-inferiority" could be the relevant question to ask), there is equivalence testing (to prove that your new drug is equivalent to a competitor) or non-inferiority testing (to prove that it is equivalent or better).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.