# Calculating statistical significance with unequal sample sizes and unequal variances

I have two samples, one with $n_1 = 41,000$ and the other with $n_2 = 881$; the larger sample has a standard deviation of $13.74$, and the smaller has an $SD=10.75$. The means are different, and when I run a Welch's t-test, I get a $p < .001$. I'm not sure if that's the appropriate test. I checked the skew for both samples; it was $29$ for the large sample and $9$ for the small one. Should I use a Mann-Whitney U test, or do I have enough data to assume normally distributed samples? In the end I need to know if the means of the samples are statistically different and be able to say one mean is $X$ times larger than the other.

-
SDs of 14 & 11 are not that much different to likely cause large problems. Skew is typically a bigger deal, but 10 may not be too big given that even your 'smaller' group is very large. Are the skews in the same direction? Are the skews similar in magnitude? –  gung Feb 25 '14 at 1:38
The skews are 29 for the large set and 9 for the smaller set. Although I've read that if the sample size is large enough (i.e >100) you could assume that the sample is normally distributed even if the population isn't. Not sure how applicable that is here. –  user1112641 Feb 25 '14 at 18:53
It doesn't mean "you could assume that the sample is normally distributed even if the population isn't", it might mean you could assume that the sampling distribution is normally distributed even though the sample (& presumably the population) isn't. –  gung Feb 25 '14 at 20:05

1) Those standard deviations aren't so badly different.

2) Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.

You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.

3) The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10' ... but that doesn't really say how big they are. If, say the skewness in the smaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)

--

The Welch test should be okay.

Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.

Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.

The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.

For example, I used simulation in parts of my answer to this related question. I carried out those simulations in R.

So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.

-
Thanks! Is there a way to quantify the not so bad-ness in the difference of two variances? –  user1112641 Feb 26 '14 at 19:15
I've added a little to the end of my answer –  Glen_b Feb 26 '14 at 22:37

You nay want to try conducting a Levene's test, this checks the assumption of homogeneous variance. If the p value is $< .25$, you assume unequal variances, and your safest bet is to run welch related tests. If your p value is $>.25$, run ANOVA, which for two groups is just a t-test. The assumption of normality is not a big deal when your sample size is so large.

Note that the p value of .25 for the Levene's test is a bit arbitrary, other books/professors may suggest a less conservative value such as .15 or .10, etc...

-
W/ the reported N's, there is no point in running Levene's test. It will be significant. The larger point, however, is that the reported difference in variances will have little impact on the results. The question is Welch's t-test vs Mann-Whitney U-test, not Welch's t-test vs regular t-test. –  gung Feb 25 '14 at 1:53
@gung Fair enough, I am assuming that the Welch t-test would do given that the question asked for a comparison of the means. I believe Mann-Whitney U-test compares medians, but I am not certain. I guess I would not be able to look at the sample sizes and know that the Levene's test will be significant. The only only quick hand way I know of to check homogeneity of variances is if $\dfrac{\sigma_1}{\sigma_2}<4$, then the assumption holds. –  k6adams Feb 25 '14 at 2:24
That's a standard (& reasonable) rule of thumb. (Technically, it's $\frac{\sigma_1^2}{\sigma_2^2}<4$, though.) Since that holds here, we needn't be too concerned, especially since the small group is 881. –  gung Feb 25 '14 at 2:28
AH, I will correct that. –  k6adams Feb 25 '14 at 2:30
Nevermind, it won't let me edit the comment, I will leave it up there to give the conversation context. –  k6adams Feb 25 '14 at 2:31