4
$\begingroup$

If I have two groups, one with a sample size of, say, 700,000 observations and another with 10,000 observations and I want to test the difference between the means of the two groups, what would be the best way to go about it?

  1. Using Welch's t-test because it is not affected by unequal variances (which usually show up because of the difference in sample sizes).
  2. Taking a random sample from the '700,000' group? (a random sample of 10k observations). I took 1000 samples of 10k from the bigger group and the p-value was always <0.05. But another interesting thing I read somewhere that p-values are always low if the data sample size is really big.
  3. Any better way of doing it?

Also, will the Welch's t-test results be untrustworthy because of the underlying skewed distributions?

$\endgroup$
5
  • 1
    $\begingroup$ Not true that the P-value is always low for large sample sizes. If there really is a difference, a large sample will increase the probability of detecting that. But if there is no difference, a large sample won't 'invent' one for you. Example in R: set.seed(12); x = rnorm(1000,100,10); y = rnorm(1000,100,12) t.test(x,y)$p.val returns P-value 0.1664101. $\endgroup$ – BruceET Aug 8 '20 at 0:18
  • 1
    $\begingroup$ 1) Why do you say that the unequal variance shows up due to unequal sample sizes? 2) Do you mean Welch’s t-test as opposed to the equal- variance t-test? $\endgroup$ – Dave Aug 8 '20 at 1:28
  • $\begingroup$ @BruceET Thanks for the example, I shouldn't have stated it like a fact. What I meant by that was that because of the larger sample size the test would be sensitive to the smallest of difference. Maybe using something like Cohen's d would help look at the extent of the effect? $\endgroup$ – Vardayini Aug 9 '20 at 8:25
  • $\begingroup$ @Dave 1) I'm a newbie, so I read on a lot of answers that the assumption of equal sample sizes is there to say that approx. the variances are equal in the two groups. 2) Yes, I mean Welch's t-test. my bad, I'll update the question. $\endgroup$ – Vardayini Aug 9 '20 at 8:28
  • $\begingroup$ Your last paragraph seems to be the only place you refer to "underlying skewed distributions". With pronounced skewness, even whether t tests of any flavour are a good idea could be at issue. Other way round, if skewness here is a slip for unequal spread, please fix the question. $\endgroup$ – Nick Cox Aug 9 '20 at 10:17
3
$\begingroup$

If you have data on $n_1 = 700,000$ in Group 1 and $n_2= 10,000,$ then I wonder about two issues:

(a) Unbiasedness. Were the observations randomly taken in order to represent the groups fairly? Or are they self-selected subjects who may not be representative. On the positive side, are these samples so large that they essentially exhaust their respective populations--perhaps making issues of sampling bias are less important.

(b) Descriptive or testing approach. With such large samples, it may be sufficient to show summary statistics, data tables, or graphical descriptions of the data. If you feel testing is important, then what would be the point of taking a subsample of the larger group? Doing that to "even up" the sample sizes is not necessary because test accommodate to unequal sample sizes. Doing that to improve "randomness" is futile: if the large sample is unrepresentative of the population, then a small subsample can be no better.

If data in the two groups are approximately normal, then a Welch two-sample t test with the sample sizes $n_1$ and $n_2$ will not be spoiled by unequal sample sizes or by unequal population variances. As mentioned above test results may not tell you anything you don't already know from descriptive statistics, but the test procedure itself should introduce no fresh difficulties.

You briefly mention that the data are skewed. Without further information it is difficult to say whether skewness would be invalidate the t test even with these large sample sizes. (If skewness is severe and is similar between the two distributions, it may be better to use a two-sample Wilcoxon (rank sum) test. Due to lack of information, I am ignoring this issue for now.)

Here are two simulated datasets of sizes $n_1$ and $n_2$ with a small, but noticeable difference in means and unequal variances.

set.seed(2020)
x1 = rnorm(700000, 103, 15)
x2 = rnorm(10000,  100, 20)

summary(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  32.59   92.91  102.99  103.02  113.12  175.41 
summary(x2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  28.32   86.68  100.10   99.89  113.25  176.54 

The sample mean and median of the larger sample are larger than the sample mean and median, respectively, of the smaller sample. Boxplots show the medians, and give a clear impression that values in the larger sample are somewhat larger than those in the smaller sample. The boxplot also shows greater variability for the first sample. [Ordinarily, one would make the boxplot for the larger group thicker than the other one, but the difference seemed distracting here.]

boxplot(x1, x2, col="skyblue2", names=c(1,2), 
        pch=20, horizontal=T)

enter image description here

The test gives a reasonable answer. The P-value is very nearly $0$ so there is little question of statistical significance. Also, a 95% confidence interval $(2.74, 3.52)$ for the difference $\mu_1 - \mu_2$ in sample means is convincingly far from including $0.$

t.test(x1, x2)

        Welch Two Sample t-test

data:  x1 and x2
t = 15.771, df = 10164, p-value < 2.2e-16
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 2.740895 3.518955
sample estimates:
mean of x mean of y 
103.02070  99.89077 

Note: A Wilcoxon rank sum test also shows significance for my simulated data:

wilcox.test(x1, x2)$p.val
[1] 1.130024e-64
$\endgroup$
14
  • 1
    $\begingroup$ (+1) Careful, well-crafted answer as always. The boxplot works well in your example but it's worth underlining that it shows medians and quartiles, not any quantities directly relevant to a t-test of any flavour. $\endgroup$ – Nick Cox Aug 9 '20 at 10:14
  • 2
    $\begingroup$ The Wilcoxon test tests a more general hypothesis of whether values from one population tend to be larger than values from the other population. When such stochastic ordering is of interest, then use the Wilcoxon test and don't worry so much about variances or skewness. Or consider the Kolmogorov-Smirnov two-sample test (difference in two cumulative distribution functions). In general, if you want to go parametric, you may need a 4-parameter skew t-distribution for your data. $\endgroup$ – Frank Harrell Aug 9 '20 at 10:53
  • 1
    $\begingroup$ (a) Buying or not buying is hard to model. Depends on circumstances (hand sanitizer in pandemic), season (wooly sox in winter), trendiness (black nail polish--who knows why or for how long). Careful description may be best. (b) Nuking far outliers is almost always wrong. For I-commerce they might be major story. (c) Right that Wilcoxon test is not just a t test for when you worry about non-normal data. @FrankHarrell's suggestion on K-S test worth considering, but possibly challenging to implement / interpret With whatever test, think hard about what its saying not just what you want to know. $\endgroup$ – BruceET Aug 9 '20 at 13:15
  • 1
    $\begingroup$ Just don't say that the t-test just needs the means to be approximately normally distributed. That's not nearly enough. $\endgroup$ – Frank Harrell Aug 14 '20 at 21:26
  • 1
    $\begingroup$ There are several posts on these issues on this site. These are very common misconceptions. If you have a log normal distribution the CLT may be bogus for up to n=50,000. Read papers by Rand Wilcox. $\endgroup$ – Frank Harrell Aug 14 '20 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.