When the sample size is large, the results tend to be statistically significant even when there is a small difference between the means of two groups. however, sometimes we see the results are not significant, despite the fact that the difference between the means of two groups is small with a large sample size. Can anyone describe its reason within a simple approach? Many thanks

  • 2
    $\begingroup$ It also depends on the variability in the groups. $\endgroup$ – Sal Mangiafico Apr 21 at 21:17

The explanation is in the formula for the t statistic:

$t = \frac{m_a-m_b}{\sqrt{S^2/n_A + S^2/n_b}}$

The numerator is the difference in means. The denominator is the variation in the groups. So, you can have different t statistics for the same difference in means with the same sample size, depending on how much variation there is in each group.

If you would like a very intuitive explanation with no math, I wrote one about ANOVA (and a t-test is an ANOVA with only two groups and one IV). Very briefly, the t test compares variation between groups to variation within groups.

  • $\begingroup$ Is that the Welch statistic you're discussing? $\endgroup$ – Glen_b Apr 22 at 17:18
  • $\begingroup$ @Glen_b, Yes, the Welch statistic. it is very useful if you give me an example using R $\endgroup$ – user3546966 Apr 22 at 22:12
  • 1
    $\begingroup$ Peter has done an excellent job identifying the intent then; I could not discern it from your question. You seek an example of what, exactly? Note that there are examples of using t-tests in R (whether Welch or ordinary equal-variance t-test or one-sample t-test / paired t-test in the help - see ?t.test and scroll down) $\endgroup$ – Glen_b Apr 23 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.