2
$\begingroup$

I am trying to determine if there is a mean difference between two test groups. The raw data (continuous scores) is distributed according to a long tail distribution.

Each test group has more than 10K observations.

After a lot of reading I concluded that I can probably use the Welch t-test to calculate a 95% confidence interval for the mean difference.

Is this correct or am I violating any important assumptions by doing that?

Cheers, Marcus

Improve this question
$\endgroup$
2
  • $\begingroup$ Do both groups show the same long tail? Do you really need a test of the mean, or just that one tends to be higher than the other? $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 13, 2016 at 17:31
  • $\begingroup$ Yes, both groups' distributions are from the same family of distributions. A confidence interval of mean or median difference would be preferable (so that I can say something about the effect size). $\endgroup$
    – Marcus
    Commented Apr 14, 2016 at 6:00

2 Answers 2

Reset to default
1
$\begingroup$

As with other T-tests, Welch's T-test hinges on the use of the central limit theorem to ensure that the sample means of the two samples are (approximately) normally distributed. This is what gets us to the (approximate) T-distribution for the test statistic. Now, it is not necessary for your samples to be normally distributed, but they must meet the requirements of one of the central limit theorems so that the sample means are normally distributed. (See further discussion on this issue in comments below.)

To meet the requirements of the classical CLT (the Lindberg-Lévy theorem) the two populations must each have finite variance, and this would exclude some long-tailed distributions. Finite variance requires that the tails of the distribution each decrease faster than cubic decay. To check if your distributions have tails that decrease faster than cubic decay (and therefore have finite variance) you should construct a tail plot for each sample, and compare this to a line showing cubic decay; see e.g., this related answer. If you are unable to establish the conditions for the CLT, and it looks like they might not hold, then the requirements of the T-test may not be valid, and the test may give poor results.

Improve this answer
$\endgroup$
9
  • $\begingroup$ The (Welch-)$t$-test does assume normally distributed populations as @Glen_b explains here. The Wikipedia page you linked to also states this. $\endgroup$
    – COOLSerdash
    Commented Jun 28, 2019 at 5:32
  • 1
    $\begingroup$ They are only independent in that case, but their covariance in the general case is $\mathbb{C}(\bar{X}, S^2) = \gamma \sigma^2 / n$, which approaches zero as $n \rightarrow \infty$ (so long as $\gamma < \infty$). Now, obviously being asymptotically uncorrelated is weaker than independence, but it is a decent first-order approximation for large $n$. $\endgroup$
    – Ben
    Commented Jun 28, 2019 at 6:10
  • 1
    $\begingroup$ Very interesting, thanks. $\endgroup$
    – COOLSerdash
    Commented Jun 28, 2019 at 6:12
  • 1
    $\begingroup$ Correction to comment above - that should be $\mathbb{C}(\bar{X},S^2) = \gamma \sigma^3/n$ (i.e., the sigma should be cubed, not squared). $\endgroup$
    – Ben
    Commented Jun 28, 2019 at 6:21
  • 1
    $\begingroup$ Yes, that sounds about right - I would think that deviations in lower-order moments would be more of a problem than deviations in higher-order moments. $\endgroup$
    – Ben
    Commented Jun 28, 2019 at 6:46
-1
$\begingroup$

This video shows a clear explanation and simple explanation of comparing two means. Three assumptions are made for this tool:

  1. Homogeneity of variance

  2. Normal distribution of populations (according to you, this is your case)

  3. Each value is sampled independently

If you can assume those three:yes, you can use Welch's t-test or student's t-test

Improve this answer
$\endgroup$
2
  • $\begingroup$ It is the second assumption that I am not sure I understand: "Normal distribution of populations (according to you, this is your case)". Raw data is definitively not normally distributed - it follows a long tail distribution (the same family of distributions for both groups). $\endgroup$
    – Marcus
    Commented Apr 14, 2016 at 6:02
  • $\begingroup$ Is it the raw data or the sample test statistic that needs to be normally distributed? (my groups are large and after running simulations I can see that my test statistic, i.e. the sampled mean difference, is indeed normally distributed) $\endgroup$
    – Marcus
    Commented Apr 14, 2016 at 6:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.