Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group differences.

Question:

  • What is a good way to test whether there are no group differences?
  • How would you determine the sample size needed to adequately test for no group differences?

Initial Thoughts:

  • It would not be enough to do a standard t-test because a failure to reject the null hypothesis does not mean that the parameter of interest is equal or close to zero; this is particularly the case with small samples.
  • I could look at the 95% confidence interval and check that all values are within a sufficiently small range; perhaps plus or minus 0.3 standard deviations.
share|improve this question
    
what do you mean by "this assumes the null hypothesis to be true" ? –  robin girard Sep 24 '10 at 8:39
    
If you want to be able to control the probability of declaring wrongly "there is a difference" you need to separate the two hypothesis (did I already mentionned I love this quote: stats.stackexchange.com/questions/726/… ;) ) –  robin girard Sep 24 '10 at 8:40
    
@Robin the p value of a null hypothesis significance test is the probability of seeing as or more extreme data than that observed assuming the null hypothesis is true; but perhaps I could word the statement above better. –  Jeromy Anglim Sep 24 '10 at 10:26
    
@Robin I modified the question to try to make my point clearer –  Jeromy Anglim Sep 24 '10 at 10:32
add comment

8 Answers 8

I think you are asking about testing for equivalence. Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent. That decision defines the 95% (or other) confidence interval limits, and sample size calculations are made on this basis.

There is a whole book on the topic.

A very common clinical "equivalent" of equivalence tests is a non-inferiority test/ trial. In this case you "prefer" one group over the other (an established treatment) and design your test to show that the new treatment is not inferior to the established treatment at some level of statistical evidence.

I think I need to credit Harvey Motulsky for the GraphPad.com site (under "Library").

share|improve this answer
add comment

Following Thylacoleo's answer, I did a little research.

The equivalence package in R has the tost() function.

See Robinson and Frose (2004) "Model validation using equivalence tests" for more info.

share|improve this answer
    
Thanks for the link and the pointer to the equivalence package. –  chl Sep 26 '10 at 9:59
add comment

Besides the already mentioned possibility of some kind of equivalence test, of which most of them, to the best of my knowledge, are mostly routed in the good old frequentist tradition, there is the possibility of conducting tests which really provide a quantification of evidence in favor of a null-hyptheses, namely bayesian tests.

An implementation of a bayesian t-test can be found here: Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t-test. Psychonomic Bulletin & Review, 16, 752-760.

There is also a tutorial on how to do all this in R:

http://www.ruudwetzels.com/index.php?src=SDtest


An alternative (perhaps more modern approach) of a Bayesian t-test is provided (with code) in this paper by Kruschke:

Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573–603. doi:10.1037/a0029146


All props for this answer (before the addition of Kruschke) should go to my colleague David Kellen. I stole his answer from this question.

share|improve this answer
    
I was wondering if someone would provide a Bayesian approach. Excellent. Thanks. –  Jeromy Anglim Sep 24 '10 at 11:41
add comment

There are a few papers I know of that could be helpful to you:

Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386. (FREE PDF)

And a correction:
Tryon, W. W., & Lewis, C. (2008). An Inferential Confidence Interval Method of Establishing Statistical Equivalence That Corrects Tryon’s (2001) Reduction Factor. Psychological Methods, 13, 272-278. (FREE PDF)

Furthermore:

Seaman, M. A. & Serlin, R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, Vol 3(4), 403-411.

share|improve this answer
    
There are tons of papers and even books on this topic. –  Michael Chernick May 6 '12 at 4:53
add comment

In the medical sciences, it is preferable to use a confidence interval approach as opposed to two one-sided tests (tost). I also recommend graphing the point estimates, CIs, and a priori-determined equivalence margins to make things very clear.

Your question would likely be addressed by such an approach.

The CONSORT guidelines for non-inferiority/equivalence studies are quite useful in this regard.

See Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ, and CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006, Mar 8;295(10):1152-60. (Link to full text.)

share|improve this answer
    
I wouldn't necessarily say that confidence intervals are preferred. In fact the confidence intervals correspond to hypothesis tests. TOST can be achieved by looking at the confidence intervals obtained by intersecting the two one sided confidence intervals that correspond to the two one-sided t tests that are used in the procedure. –  Michael Chernick May 6 '12 at 4:59
add comment

Yes. This is equivalence testing. Basically you reverse the null and alternative hypothesis and base the sample size on the power to show that the difference of the means is within the window of equivalence. Blackwelder called it "Proving the null hypothesis." This is commonly done in pharmaceutical clinical trials where equivalence of a generic drug to the marketed drug is tested or an approved drug is compared to a new formulation (often called bioequivalence). The one sided version is called non-inferiority. Some times a drug can be approved by just showing that the new drug is not inferior to the marketed competitor. Shao and Pigeot have developed a consistent bootstrap approach to bioequivalence using crossover designs.

share|improve this answer
add comment

I have recently thought about an alternative way of "equivalence testing" based on a distance between the two distributions rather than between their means.

There are some methods providing confidence intervals for the overlap of two Gaussian distributions: enter image description here

The overlap $O(P_1,P_2)$ of (between?) two distributions $P_1$ and $P_2$ has a nice probabilistic interpretation: $$1-O(P_1,P_2)= TV(P_1,P_2)$$ where $TV(P_1,P_2) = \sup_A \big|P_1(A) - P_2(A) \big|$ is the total variation distance between $P_1$ and $P_2$.

That means that, for example, if $O(P_1,P_2)>0.9$ then the probabilities given by $P_1$ and $P_2$ of any event do not differ more than $0.1$. Roughly speaking, the two distribution make the same predictions up to $10\%$.

Thus, instead of using an acceptance criterion based on a critical value for the difference between the means $\mu_1$ and $\mu_2$, as in classical equivalence testing, we could base it on a critical value for the difference between the probabilities of the predictions given by the two distributions.

I think there's an advantage in terms of "objectiveness" of the criterion. The critical value of $|\mu_1 - \mu_2|$ should be given by an expert of the real problem: this should be a value beyond which the difference has a practical importance. But sometimes nobody has a solid knowledge about the real problem and there's no expert able to provide a critical value. Adopting a conventional critical value about $TV(P_1,P_2)$ could be a way to a criterion not depending on the physical problem under consideration.

In the Gaussian case with same variances, the overlap is one-to-one related to the standardized mean difference $\frac{|\mu_1-\mu_2|}{\sigma}$.

share|improve this answer
add comment

Bootstrap differences (e.g. the difference between the means) between the 2 sample groups and check for statistical significance. A more detailed description of this approach, albeit in a different context, can be found here http://www.automated-trading-system.com/a-different-application-of-the-bootstrap/

share|improve this answer
    
You are conflating the fallacy of accepting the null hypothesis of no difference and finding evidence that two quantities are equivalent. –  Alexis May 4 at 15:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.