Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group differences.


  • What is a good way to test whether there are no group differences?
  • How would you determine the sample size needed to adequately test for no group differences?

Initial Thoughts:

  • It would not be enough to do a standard t-test because a failure to reject the null hypothesis does not mean that the parameter of interest is equal or close to zero; this is particularly the case with small samples.
  • I could look at the 95% confidence interval and check that all values are within a sufficiently small range; perhaps plus or minus 0.3 standard deviations.
  • what do you mean by "this assumes the null hypothesis to be true" ? – robin girard Sep 24 '10 at 8:39
  • If you want to be able to control the probability of declaring wrongly "there is a difference" you need to separate the two hypothesis (did I already mentionned I love this quote:… ;) ) – robin girard Sep 24 '10 at 8:40
  • @Robin the p value of a null hypothesis significance test is the probability of seeing as or more extreme data than that observed assuming the null hypothesis is true; but perhaps I could word the statement above better. – Jeromy Anglim Sep 24 '10 at 10:26
  • @Robin I modified the question to try to make my point clearer – Jeromy Anglim Sep 24 '10 at 10:32

I think you are asking about testing for equivalence. Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent. That decision defines the 95% (or other) confidence interval limits, and sample size calculations are made on this basis.

There is a whole book on the topic.

A very common clinical "equivalent" of equivalence tests is a non-inferiority test/ trial. In this case you "prefer" one group over the other (an established treatment) and design your test to show that the new treatment is not inferior to the established treatment at some level of statistical evidence.

I think I need to credit Harvey Motulsky for the site (under "Library").

Besides the already mentioned possibility of some kind of equivalence test, of which most of them, to the best of my knowledge, are mostly routed in the good old frequentist tradition, there is the possibility of conducting tests which really provide a quantification of evidence in favor of a null-hyptheses, namely bayesian tests.

An implementation of a bayesian t-test can be found here: Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t-test. Psychonomic Bulletin & Review, 16, 752-760.

There is also a tutorial on how to do all this in R:

An alternative (perhaps more modern approach) of a Bayesian t-test is provided (with code) in this paper by Kruschke:

Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573–603. doi:10.1037/a0029146

All props for this answer (before the addition of Kruschke) should go to my colleague David Kellen. I stole his answer from this question.

  • I was wondering if someone would provide a Bayesian approach. Excellent. Thanks. – Jeromy Anglim Sep 24 '10 at 11:41
  • 1
    It may be worth updating this answer to include a reference to the awesome BayesFactor package for R. – crsh Apr 27 '17 at 16:22

Following Thylacoleo's answer, I did a little research.

The equivalence package in R has the tost() function.

See Robinson and Frose (2004) "Model validation using equivalence tests" for more info.

  • Thanks for the link and the pointer to the equivalence package. – chl Sep 26 '10 at 9:59

There are a few papers I know of that could be helpful to you:

Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386. (FREE PDF)

And a correction:
Tryon, W. W., & Lewis, C. (2008). An Inferential Confidence Interval Method of Establishing Statistical Equivalence That Corrects Tryon’s (2001) Reduction Factor. Psychological Methods, 13, 272-278. (FREE PDF)


Seaman, M. A. & Serlin, R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, Vol 3(4), 403-411.

  • There are tons of papers and even books on this topic. – Michael Chernick May 6 '12 at 4:53

I have recently thought about an alternative way of "equivalence testing" based on a distance between the two distributions rather than between their means.

There are some methods providing confidence intervals for the overlap of two Gaussian distributions: enter image description here

The overlap $O(P_1,P_2)$ of (between?) two distributions $P_1$ and $P_2$ has a nice probabilistic interpretation: $$1-O(P_1,P_2)= TV(P_1,P_2)$$ where $TV(P_1,P_2) = \sup_A \big|P_1(A) - P_2(A) \big|$ is the total variation distance between $P_1$ and $P_2$.

That means that, for example, if $O(P_1,P_2)>0.9$ then the probabilities given by $P_1$ and $P_2$ of any event do not differ more than $0.1$. Roughly speaking, the two distributions make the same predictions up to $10\%$.

Thus, instead of using an acceptance criterion based on a critical value for the difference between the means $\mu_1$ and $\mu_2$, as in classical equivalence testing, we could base it on a critical value for the difference between the probabilities of the predictions given by the two distributions.

I think there's an advantage in terms of "objectiveness" of the criterion. The critical value of $|\mu_1 - \mu_2|$ should be given by an expert of the real problem: this should be a value beyond which the difference has a practical importance. But sometimes nobody has a solid knowledge about the real problem and there's no expert able to provide a critical value. Adopting a conventional critical value about $TV(P_1,P_2)$ could be a way to a criterion not depending on the physical problem under consideration.

In the Gaussian case with same variances, the overlap is one-to-one related to the standardized mean difference $\frac{|\mu_1-\mu_2|}{\sigma}$.

  • Do you have any resources showing overlap being used in some real problems? This sounds incredibly promising, but it's not clear to me how one would apply it in a real problem (where your conclusions are potentially several steps removed from "this distribution is pretty similar to X", thereby making it a little hard to see how that 10% TV translates into size of impact on inferences). – Stumpy Joe Pete Jul 8 '15 at 7:16
  • 1
    @StumpyJoePete I have written something in the same spirit on my blog:… – Stéphane Laurent Jul 8 '15 at 8:15

In the medical sciences, it is preferable to use a confidence interval approach as opposed to two one-sided tests (tost). I also recommend graphing the point estimates, CIs, and a priori-determined equivalence margins to make things very clear.

Your question would likely be addressed by such an approach.

The CONSORT guidelines for non-inferiority/equivalence studies are quite useful in this regard.

See Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ, and CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006, Mar 8;295(10):1152-60. (Link to full text.)

  • 1
    I wouldn't necessarily say that confidence intervals are preferred. In fact the confidence intervals correspond to hypothesis tests. TOST can be achieved by looking at the confidence intervals obtained by intersecting the two one sided confidence intervals that correspond to the two one-sided t tests that are used in the procedure. – Michael Chernick May 6 '12 at 4:59

Yes. This is equivalence testing. Basically you reverse the null and alternative hypothesis and base the sample size on the power to show that the difference of the means is within the window of equivalence. Blackwelder called it "Proving the null hypothesis." This is commonly done in pharmaceutical clinical trials where equivalence of a generic drug to the marketed drug is tested or an approved drug is compared to a new formulation (often called bioequivalence). The one sided version is called non-inferiority. Some times a drug can be approved by just showing that the new drug is not inferior to the marketed competitor. Shao and Pigeot have developed a consistent bootstrap approach to bioequivalence using crossover designs.

Bootstrap differences (e.g. the difference between the means) between the 2 sample groups and check for statistical significance. A more detailed description of this approach, albeit in a different context, can be found here

  • 1
    You are conflating the fallacy of accepting the null hypothesis of no difference and finding evidence that two quantities are equivalent. – Alexis May 4 '14 at 15:00

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.