41
$\begingroup$

Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group differences.

Question:

  • What is a good way to test whether there are no group differences?
  • How would you determine the sample size needed to adequately test for no group differences?

Initial Thoughts:

  • It would not be enough to do a standard t-test because a failure to reject the null hypothesis does not mean that the parameter of interest is equal or close to zero; this is particularly the case with small samples.
  • I could look at the 95% confidence interval and check that all values are within a sufficiently small range; perhaps plus or minus 0.3 standard deviations.
Improve this question
$\endgroup$
4
  • $\begingroup$ what do you mean by "this assumes the null hypothesis to be true" ? $\endgroup$
    – robin girard
    Commented Sep 24, 2010 at 8:39
  • $\begingroup$ If you want to be able to control the probability of declaring wrongly "there is a difference" you need to separate the two hypothesis (did I already mentionned I love this quote: stats.stackexchange.com/questions/726/… ;) ) $\endgroup$
    – robin girard
    Commented Sep 24, 2010 at 8:40
  • $\begingroup$ @Robin the p value of a null hypothesis significance test is the probability of seeing as or more extreme data than that observed assuming the null hypothesis is true; but perhaps I could word the statement above better. $\endgroup$
    – Jeromy Anglim
    Commented Sep 24, 2010 at 10:26
  • $\begingroup$ @Robin I modified the question to try to make my point clearer $\endgroup$
    – Jeromy Anglim
    Commented Sep 24, 2010 at 10:32

8 Answers 8

Reset to default
21
$\begingroup$

I think you are asking about testing for equivalence. Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent. That decision defines the 95% (or other) confidence interval limits, and sample size calculations are made on this basis.

There is a whole book on the topic.

A very common clinical "equivalent" of equivalence tests is a non-inferiority test/ trial. In this case you "prefer" one group over the other (an established treatment) and design your test to show that the new treatment is not inferior to the established treatment at some level of statistical evidence.

I think I need to credit Harvey Motulsky for the GraphPad.com site (under "Library").

Improve this answer
$\endgroup$
0
17
$\begingroup$

Besides the already mentioned possibility of some kind of equivalence test, of which most of them, to the best of my knowledge, are mostly routed in the good old frequentist tradition, there is the possibility of conducting tests which really provide a quantification of evidence in favor of a null-hyptheses, namely bayesian tests.

An implementation of a bayesian t-test can be found here: Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t-test. Psychonomic Bulletin & Review, 16, 752-760.

There is also a tutorial on how to do all this in R:

http://www.ruudwetzels.com/index.php?src=SDtest

An alternative (perhaps more modern approach) of a Bayesian t-test is provided (with code) in this paper by Kruschke:

Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573–603. doi:10.1037/a0029146

All props for this answer (before the addition of Kruschke) should go to my colleague David Kellen. I stole his answer from this question.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ It may be worth updating this answer to include a reference to the awesome BayesFactor package for R. $\endgroup$
    – crsh
    Commented Apr 27, 2017 at 16:22
  • 1
    $\begingroup$ For those interested in the tutorial, the website is no longer functional. The Wayback Machine has a copy at web.archive.org/web/20090312002033/http://www.ruudwetzels.com/… $\endgroup$
    – Matthew Walker
    Commented Jun 8, 2021 at 22:01
14
$\begingroup$

Following Thylacoleo's answer, I did a little research.

The equivalence package in R has the tost() function.

See Robinson and Frose (2004) "Model validation using equivalence tests" for more info.

Improve this answer
$\endgroup$
0
8
$\begingroup$

There are a few papers I know of that could be helpful to you:

Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386. (FREE PDF)

And a correction:
Tryon, W. W., & Lewis, C. (2008). An Inferential Confidence Interval Method of Establishing Statistical Equivalence That Corrects Tryon’s (2001) Reduction Factor. Psychological Methods, 13, 272-278. (FREE PDF)

Furthermore:

Seaman, M. A. & Serlin, R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, Vol 3(4), 403-411.

Improve this answer
$\endgroup$
0
8
$\begingroup$

I have recently thought about an alternative way of "equivalence testing" based on a distance between the two distributions rather than between their means.

There are some methods providing confidence intervals for the overlap of two Gaussian distributions: enter image description here

The overlap $O(P_1,P_2)$ of (between?) two distributions $P_1$ and $P_2$ has a nice probabilistic interpretation: $$1-O(P_1,P_2)= TV(P_1,P_2)$$ where $TV(P_1,P_2) = \sup_A \big|P_1(A) - P_2(A) \big|$ is the total variation distance between $P_1$ and $P_2$.

That means that, for example, if $O(P_1,P_2)>0.9$ then the probabilities given by $P_1$ and $P_2$ of any event do not differ more than $0.1$. Roughly speaking, the two distributions make the same predictions up to $10\%$.

Thus, instead of using an acceptance criterion based on a critical value for the difference between the means $\mu_1$ and $\mu_2$, as in classical equivalence testing, we could base it on a critical value for the difference between the probabilities of the predictions given by the two distributions.

I think there's an advantage in terms of "objectiveness" of the criterion. The critical value of $|\mu_1 - \mu_2|$ should be given by an expert of the real problem: this should be a value beyond which the difference has a practical importance. But sometimes nobody has a solid knowledge about the real problem and there's no expert able to provide a critical value. Adopting a conventional critical value about $TV(P_1,P_2)$ could be a way to a criterion not depending on the physical problem under consideration.

In the Gaussian case with same variances, the overlap is one-to-one related to the standardized mean difference $\frac{|\mu_1-\mu_2|}{\sigma}$.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Do you have any resources showing overlap being used in some real problems? This sounds incredibly promising, but it's not clear to me how one would apply it in a real problem (where your conclusions are potentially several steps removed from "this distribution is pretty similar to X", thereby making it a little hard to see how that 10% TV translates into size of impact on inferences). $\endgroup$
    – Stumpy Joe Pete
    Commented Jul 8, 2015 at 7:16
  • 2
    $\begingroup$ @StumpyJoePete I have written something in the same spirit on my blog: stla.github.io/stlapblog/posts/… $\endgroup$
    – Stéphane Laurent
    Commented Jul 8, 2015 at 8:15
5
$\begingroup$

In the medical sciences, it is preferable to use a confidence interval approach as opposed to two one-sided tests (tost). I also recommend graphing the point estimates, CIs, and a priori-determined equivalence margins to make things very clear.

Your question would likely be addressed by such an approach.

The CONSORT guidelines for non-inferiority/equivalence studies are quite useful in this regard.

See Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ, and CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006, Mar 8;295(10):1152-60. (Link to full text.)

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I wouldn't necessarily say that confidence intervals are preferred. In fact the confidence intervals correspond to hypothesis tests. TOST can be achieved by looking at the confidence intervals obtained by intersecting the two one sided confidence intervals that correspond to the two one-sided t tests that are used in the procedure. $\endgroup$
    – Michael R. Chernick
    Commented May 6, 2012 at 4:59
5
$\begingroup$

Yes. This is equivalence testing. Basically you reverse the null and alternative hypothesis and base the sample size on the power to show that the difference of the means is within the window of equivalence. Blackwelder called it "Proving the null hypothesis." This is commonly done in pharmaceutical clinical trials where equivalence of a generic drug to the marketed drug is tested or an approved drug is compared to a new formulation (often called bioequivalence). The one sided version is called non-inferiority. Some times a drug can be approved by just showing that the new drug is not inferior to the marketed competitor. Shao and Pigeot have developed a consistent bootstrap approach to bioequivalence using crossover designs.

Improve this answer
$\endgroup$
0
$\begingroup$

Bootstrap differences (e.g. the difference between the means) between the 2 sample groups and check for statistical significance. A more detailed description of this approach, albeit in a different context, can be found here http://www.automated-trading-system.com/a-different-application-of-the-bootstrap/

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You are conflating the fallacy of accepting the null hypothesis of no difference and finding evidence that two quantities are equivalent. $\endgroup$
    – Alexis
    Commented May 4, 2014 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.