How to test hypothesis of no group differences? Imagine you have a study with two groups (e.g., males and females) looking at a numeric dependent variable (e.g., intelligence test scores) and you have the hypothesis that there are no group differences.
Question:


*

*What is a good way to test whether there are no group differences?

*How would you determine the sample size needed to adequately test for no group differences?


Initial Thoughts: 


*

*It would not be enough to do a standard t-test because a failure to reject the null hypothesis does not mean that the parameter of interest is equal or close to zero; this is particularly the case with small samples.

*I could look at the 95% confidence interval and check that all values are within a sufficiently small range; perhaps plus or minus 0.3 standard deviations.

 A: There are a few papers I know of that could be helpful to you:
Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386.  (FREE PDF)
And a correction:
Tryon, W. W., & Lewis, C. (2008). An Inferential Confidence Interval Method of Establishing Statistical Equivalence That Corrects Tryon’s (2001) Reduction Factor. Psychological Methods, 13,  272-278. (FREE PDF)
Furthermore:
Seaman, M. A. & Serlin, R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, Vol 3(4), 403-411.
A: I have recently thought about an alternative way of "equivalence testing" based on a distance between the two distributions rather than between their means. 
There are some methods providing confidence intervals for the overlap of two Gaussian distributions: 
The overlap $O(P_1,P_2)$ of (between?) two distributions $P_1$ and $P_2$ has a nice probabilistic interpretation: $$1-O(P_1,P_2)= TV(P_1,P_2)$$ 
where $TV(P_1,P_2) = \sup_A \big|P_1(A) - P_2(A) \big|$ is the total variation distance between $P_1$ and $P_2$. 
That means that, for example, if $O(P_1,P_2)>0.9$ then the probabilities given by $P_1$ and $P_2$ of any event do not differ more than $0.1$. Roughly speaking, the two distributions make the same predictions up to $10\%$.
Thus, instead of using an acceptance criterion based on a critical value for the difference between the means $\mu_1$ and $\mu_2$, as in classical equivalence testing, we could base it on a critical value for the difference between the probabilities of the predictions given by the two distributions. 
I think there's an advantage in terms of "objectiveness" of the criterion. The critical value of $|\mu_1 - \mu_2|$ should be given by an expert of the real problem: this should be a value beyond which the difference has a practical importance. But sometimes nobody has a solid knowledge about the real problem and there's no expert able to provide a critical value. Adopting a conventional critical value about $TV(P_1,P_2)$ could be a way to a criterion not depending on the physical problem under consideration.
In the Gaussian case with same variances, the overlap is one-to-one related to the standardized mean difference $\frac{|\mu_1-\mu_2|}{\sigma}$.
A: In the medical sciences, it is preferable to use a confidence interval approach as opposed to two one-sided tests (tost). I also recommend graphing the point estimates, CIs, and a priori-determined equivalence margins to make things very clear.
Your question would likely be addressed by such an approach.
The CONSORT guidelines for non-inferiority/equivalence studies are quite useful in this regard.
See Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ, and CONSORT Group. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006, Mar 8;295(10):1152-60. (Link to full text.)
A: Yes.  This is equivalence testing.  Basically you reverse the null and alternative hypothesis and base the sample size on the power to show that the difference of the means is within the window of equivalence.  Blackwelder called it "Proving the null hypothesis." This is commonly done in pharmaceutical clinical trials where equivalence of a generic drug to the marketed drug is tested or an approved drug is compared to a new formulation (often called bioequivalence).  The one sided version is called non-inferiority.  Some times a drug can be approved by just showing that the new drug is not inferior to the marketed competitor.  Shao and Pigeot have developed a consistent bootstrap approach to bioequivalence using crossover designs.
A: I think you are asking about testing for equivalence. Essentially you need to decide how large a difference is acceptable for you to still conclude that the two groups are effectively equivalent. That decision defines the 95% (or other) confidence interval limits, and sample size calculations are made on this basis. 
There is a whole book on the topic.
A very common clinical "equivalent" of equivalence tests is a non-inferiority test/ trial. In this case you "prefer" one group over the other (an established treatment) and design your test to show that the new treatment is not inferior to the established treatment at some level of statistical evidence.
I think I need to credit Harvey Motulsky for the GraphPad.com site (under "Library").
A: Besides the already mentioned possibility of some kind of equivalence test, of which most of them, to the best of my knowledge, are mostly routed in the good old frequentist tradition, there is the possibility of conducting tests which really provide a quantification of evidence in favor of a null-hyptheses, namely bayesian tests.
An implementation of a bayesian t-test can be found here:
Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t-test. Psychonomic Bulletin & Review, 16, 752-760.
There is also a tutorial on how to do all this in R:
http://www.ruudwetzels.com/index.php?src=SDtest

An alternative (perhaps more modern approach) of a Bayesian t-test is provided (with code) in this paper by Kruschke:
Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573–603. doi:10.1037/a0029146

All props for this answer (before the addition of Kruschke) should go to my colleague David Kellen. I stole his answer from this question.
A: Following Thylacoleo's answer, I did a little research.
The equivalence package in R has the tost() function.
See Robinson and Frose (2004) "Model validation using equivalence tests" for more info.
A: Bootstrap differences (e.g. the difference between the means) between the 2 sample groups and check for statistical significance. A more detailed description of this approach, albeit in a different context, can be found here http://www.automated-trading-system.com/a-different-application-of-the-bootstrap/
