I have two samples, one with $n_1 = 41,000$ and the other with $n_2 = 881$; the larger sample has a standard deviation of $13.74$, and the smaller has an $SD=10.75$. The means are different, and when I run a Welch's t-test, I get a $p < .001$. I'm not sure if that's the appropriate test. I checked the skew for both samples; it was $29$ for the large sample and $9$ for the small one. Should I use a Mann-Whitney U test, or do I have enough data to assume normally distributed samples? In the end I need to know if the means of the samples are statistically different and be able to say one mean is $X$ times larger than the other.
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
1) Those standard deviations aren't so badly different.
2) Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.
You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.
3) The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10' ... but that doesn't really say how big they are. If, say the skewness in the smaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)
The Welch test should be okay.
Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.
Edit: (Answering follow-up question from comments)
Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.
The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.
For example, I used simulation in parts of my answer to this related question. I carried out those simulations in R.
So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.
You nay want to try conducting a Levene's test, this checks the assumption of homogeneous variance. If the p value is $< .25$, you assume unequal variances, and your safest bet is to run welch related tests. If your p value is $>.25$, run ANOVA, which for two groups is just a t-test. The assumption of normality is not a big deal when your sample size is so large.
Note that the p value of .25 for the Levene's test is a bit arbitrary, other books/professors may suggest a less conservative value such as .15 or .10, etc...