13
$\begingroup$

I have two samples A and B. In order to check if they are the same I want to apply a t-test and see the difference in means. One of the assumptions of the t-test is that the distribution of A and B is normal.

A colleague says:

The central limit theorem says that the means of the sample distributions is approximately normal. With this in mind you can apply t-test.

Why does this argumentation work? I thought that the underlying distributions of A and B (when plotted) should be normal. And not the values (in this case means) on which the t-test is actually applied for. What does the t-test assume then? Should the data be normally distributed or the means?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
10
$\begingroup$

Okay, a few things:

1) A two-sample t-test does not assume the distributions of groups A and B are the same under the null hypothesis, even if the underlying distributions are both normal. That can only occur if you assume the standard deviations are the same, which is a hefty assumption to have. The two-sample t-test tests whether, under the null, the means are the same of the two groups. But yes, the classical two-sample t-test assumes the underlying data is normally distributed. This is the case because you do not only need the numerator to be normally distributed, but the variances also be (a scaled version of) a $\chi^2$. That being said, the t-test is fairly robust against the assumption of normality. See here.

2) It is true under a large enough sample, the underlying distribution of the means of each group is going to be approximately normal. How good that approximation depends on the underlying distribution of each group.

The general idea is this. If $X$ and $Y$ are independent, with $X$ having mean $\mu_X$ and standard deviation $\sigma_X$ and $Y$ having mean $\mu_Y$ with standard deviation $\sigma_Y$, and the respective sample $X_1,\dots,X_n$ and $Y_1,\dots,Y_m$ are large, then you can conclude $$ \frac{\bar{X}-\bar{Y}-(\mu_X-\mu_Y)}{\sqrt{\frac{\sigma^2_X}{n}+\frac{\sigma^2_Y}{m}}}$$

Is approximately normal with mean 0 and standard deviation 1. So critical values $z_{\alpha/2}$ can be used to do testing. Also, $t_{\alpha/2,\nu}$ are going to be close to $z_{\alpha/2}$, when $\nu$ is large (which occurs if the sample sizes is large). So for large enough sample sizes, a t-test can be used.

There are ways to check for this. (The standard rule of thumb is that each group has a sample size of 30 or larger, but I am usually against those rules because there are plenty of cases where that rule fails). One way you can check it (sort of) is to create a bootstrap distribution of the mean and see.

3) You can do better than approximate tests though. When you are testing to see if the means differ, your real question is really to see if the locations differ. A test that will be correct (almost) all the time will be the Mann Whitney U test. This does not test whether the means differ, but rather if the medians differ. In other words, it again tests whether one location differs from another. It may be a better option, and has a pretty high power overall.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ I wanted to express my confusion about the fact that "if the data is not normally distributed then we can always use the central limit theorem to justify the application of the t-test". For me it sounds tricky because if one assumption fails we always can turn to "central limit theorem says..." and still apply t-test $\endgroup$
    – Alina
    Commented Jul 1, 2017 at 17:46
  • $\begingroup$ Then the answer is you can't necessarily, although if you have a large enough sample size, it can be appropriate approximately. Although there might be problems with power. $\endgroup$
    – Josh
    Commented Jul 1, 2017 at 17:50
  • $\begingroup$ It again is probably better to use mann whitney. $\endgroup$
    – Josh
    Commented Jul 1, 2017 at 17:52
  • $\begingroup$ A bootstrap approach can always be used too. $\endgroup$
    – StatsStudent
    Commented Jul 1, 2017 at 17:57
  • $\begingroup$ I was asking mainly why we can justify the use of t-test by just applying the central limit theorem. On one hand, t-test makes assumptions about the normal distribution of the samples. If it does not hold, we can say "but the means from sample distributions are normally distributed", therefore we can apply t-test. So, the assumption of the t-test is either the normality of the data itself, or the the normality of the means. So, there are two assumptions of normality and either should be true in order to use t-test, is it correct? (I was not asking for the alternatives) $\endgroup$
    – Alina
    Commented Jul 1, 2017 at 17:58
4
$\begingroup$

Short answer: your colleague is right.

In the end, the t statistic depends only on the mean and variance of the two samples. The CLT says that (under most circumstances) those rapidly become normal even when the underlying population distribution is not.

So the t-test is quite robust to (most) departures from normality. This has been verified by many simulation studies. Note, by the way, that it is not at all robust to departures from homogeneity of variance.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What does the t-test assume then? Should the data be normally distributed or the means? So, if first does not hold, we can (almost) always apply the "central limit theorem saying"? $\endgroup$
    – Alina
    Commented Jul 1, 2017 at 18:14
  • 1
    $\begingroup$ Given the means and variances, you can run the t-test; you don't need the data anymore. This shows you that the t-test null distribution can't depend on the distribution of the data, only on the distribution of the means and variances. $\endgroup$
    – David Wright
    Commented Jul 1, 2017 at 18:18
  • 1
    $\begingroup$ The colleague is not right. Instead they are effectively assuming the standard deviations are known and are applicable to the raw data, and that the distributions are symmetric. If distributions are asymmetric, the SD is not independent of the mean and the t-distribution does not apply until N is very huge. An even larger point: if you should have taken log(Y) instead of analyzing Y and hoping the CLT rescues you, the SD is not even an appropriate quantity to compute on the raw scale. The CLT is an academic exercise and does not provide an accurate enough approximation often enough. $\endgroup$
    – Frank Harrell
    Commented Mar 23, 2019 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.