# How to test hypothesis of no group differences?

Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group differences.

Question:

• What is a good way to test whether there are no group differences?
• How would you determine the sample size needed to adequately test for no group differences?

Initial Thoughts:

• It would not be enough to do a standard t-test because a failure to reject the null hypothesis does not mean that the parameter of interest is equal or close to zero; this is particularly the case with small samples.
• I could look at the 95% confidence interval and check that all values are within a sufficiently small range; perhaps plus or minus 0.3 standard deviations.
• what do you mean by "this assumes the null hypothesis to be true" ? Sep 24, 2010 at 8:39
• If you want to be able to control the probability of declaring wrongly "there is a difference" you need to separate the two hypothesis (did I already mentionned I love this quote: stats.stackexchange.com/questions/726/… ;) ) Sep 24, 2010 at 8:40
• @Robin the p value of a null hypothesis significance test is the probability of seeing as or more extreme data than that observed assuming the null hypothesis is true; but perhaps I could word the statement above better. Sep 24, 2010 at 10:26
• @Robin I modified the question to try to make my point clearer Sep 24, 2010 at 10:32

I think you are asking about testing for equivalence. Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent. That decision defines the 95% (or other) confidence interval limits, and sample size calculations are made on this basis.

There is a whole book on the topic.

A very common clinical "equivalent" of equivalence tests is a non-inferiority test/ trial. In this case you "prefer" one group over the other (an established treatment) and design your test to show that the new treatment is not inferior to the established treatment at some level of statistical evidence.

I think I need to credit Harvey Motulsky for the GraphPad.com site (under "Library").

Besides the already mentioned possibility of some kind of equivalence test, of which most of them, to the best of my knowledge, are mostly routed in the good old frequentist tradition, there is the possibility of conducting tests which really provide a quantification of evidence in favor of a null-hyptheses, namely bayesian tests.

An implementation of a bayesian t-test can be found here: Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t-test. Psychonomic Bulletin & Review, 16, 752-760.

There is also a tutorial on how to do all this in R:

http://www.ruudwetzels.com/index.php?src=SDtest

An alternative (perhaps more modern approach) of a Bayesian t-test is provided (with code) in this paper by Kruschke:

Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573–603. doi:10.1037/a0029146

All props for this answer (before the addition of Kruschke) should go to my colleague David Kellen. I stole his answer from this question.

• It may be worth updating this answer to include a reference to the awesome BayesFactor package for R.
– crsh
Apr 27, 2017 at 16:22
• For those interested in the tutorial, the website is no longer functional. The Wayback Machine has a copy at web.archive.org/web/20090312002033/http://www.ruudwetzels.com/… Jun 8, 2021 at 22:01

Following Thylacoleo's answer, I did a little research.

The equivalence package in R has the tost() function.

See Robinson and Frose (2004) "Model validation using equivalence tests" for more info.

There are a few papers I know of that could be helpful to you:

Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386. (FREE PDF)

And a correction:
Tryon, W. W., & Lewis, C. (2008). An Inferential Confidence Interval Method of Establishing Statistical Equivalence That Corrects Tryon’s (2001) Reduction Factor. Psychological Methods, 13, 272-278. (FREE PDF)

Furthermore:

Seaman, M. A. & Serlin, R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, Vol 3(4), 403-411.

I have recently thought about an alternative way of "equivalence testing" based on a distance between the two distributions rather than between their means.

There are some methods providing confidence intervals for the overlap of two Gaussian distributions: The overlap $O(P_1,P_2)$ of (between?) two distributions $P_1$ and $P_2$ has a nice probabilistic interpretation: $$1-O(P_1,P_2)= TV(P_1,P_2)$$ where $TV(P_1,P_2) = \sup_A \big|P_1(A) - P_2(A) \big|$ is the total variation distance between $P_1$ and $P_2$.

That means that, for example, if $O(P_1,P_2)>0.9$ then the probabilities given by $P_1$ and $P_2$ of any event do not differ more than $0.1$. Roughly speaking, the two distributions make the same predictions up to $10\%$.

Thus, instead of using an acceptance criterion based on a critical value for the difference between the means $\mu_1$ and $\mu_2$, as in classical equivalence testing, we could base it on a critical value for the difference between the probabilities of the predictions given by the two distributions.

I think there's an advantage in terms of "objectiveness" of the criterion. The critical value of $|\mu_1 - \mu_2|$ should be given by an expert of the real problem: this should be a value beyond which the difference has a practical importance. But sometimes nobody has a solid knowledge about the real problem and there's no expert able to provide a critical value. Adopting a conventional critical value about $TV(P_1,P_2)$ could be a way to a criterion not depending on the physical problem under consideration.

In the Gaussian case with same variances, the overlap is one-to-one related to the standardized mean difference $\frac{|\mu_1-\mu_2|}{\sigma}$.

• Do you have any resources showing overlap being used in some real problems? This sounds incredibly promising, but it's not clear to me how one would apply it in a real problem (where your conclusions are potentially several steps removed from "this distribution is pretty similar to X", thereby making it a little hard to see how that 10% TV translates into size of impact on inferences). Jul 8, 2015 at 7:16
• @StumpyJoePete I have written something in the same spirit on my blog: stla.github.io/stlapblog/posts/… Jul 8, 2015 at 8:15

In the medical sciences, it is preferable to use a confidence interval approach as opposed to two one-sided tests (tost). I also recommend graphing the point estimates, CIs, and a priori-determined equivalence margins to make things very clear.