I have two samples, one with $n_1 = 41,000$ and the other with $n_2 = 881$; the larger sample has a standard deviation of $13.74$, and the smaller has an $SD=10.75$. The means are different, and when I run a Welch's t-test, I get a $p < .001$. I'm not sure if that's the appropriate test. I checked the skew for both samples; it was $29$ for the large sample and $9$ for the small one. Should I use a Mann-Whitney U test, or do I have enough data to assume normally distributed samples? In the end I need to know if the means of the samples are statistically different and be able to say one mean is $X$ times larger than the other.
$\begingroup$
$\endgroup$
3
-
$\begingroup$ SDs of 14 & 11 are not that much different to likely cause large problems. Skew is typically a bigger deal, but 10 may not be too big given that even your 'smaller' group is very large. Are the skews in the same direction? Are the skews similar in magnitude? $\endgroup$– gung - Reinstate MonicaCommented Feb 25, 2014 at 1:38
-
$\begingroup$ The skews are 29 for the large set and 9 for the smaller set. Although I've read that if the sample size is large enough (i.e >100) you could assume that the sample is normally distributed even if the population isn't. Not sure how applicable that is here. $\endgroup$– user1112641Commented Feb 25, 2014 at 18:53
-
2$\begingroup$ It doesn't mean "you could assume that the sample is normally distributed even if the population isn't", it might mean you could assume that the sampling distribution is normally distributed even though the sample (& presumably the population) isn't. $\endgroup$– gung - Reinstate MonicaCommented Feb 25, 2014 at 20:05
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
1) Those standard deviations aren't so badly different.
2) Since $n_1=41000$, and the standard deviations aren't very large, even if the variances were very different, it wouldn't matter.
You could even treat the mean of the first sample as fixed (it almost is) and do a one sample t-test.
3) The skewness likely won't matter much either, unless it's quite strong in the smaller sample. (you say 'skews above 10' ... but that doesn't really say how big they are. If, say the skewness in the smaller sample is less than 20, the distribution of the mean should still be close to normal, and between CLT for the numerator and using Slutsky's theorem for the rest of the statistic, it should be close to normal)
--
The Welch test should be okay.
Another alternative is to consider a permutation test (the standard deviations aren't all that different) or a bootstrap test. They'll likely give very similar results to what you already have.
Edit: (Answering follow-up question from comments)
Well, sure. The way to tell if the difference is not so bad is to see how much impact ignoring it would have.
The relevant measures of impact are the significance level when $H_0$ is true and power when it's false, and more generally the shape of the power function (which can reveal issues like test bias). You can most easily calculate and compare power functions under various assumptions via simulation.
For example, I used simulation in parts of my answer to this related question. I carried out those simulations in R.
So you can assume some population ratios of variances close to the one observed and see how badly it affects significance and power if you treat them as equal, and how close to the nominal significance you get if you use say the Welch approximation instead, as well as any impact on power.
-
$\begingroup$ Thanks! Is there a way to quantify the not so bad-ness in the difference of two variances? $\endgroup$ Commented Feb 26, 2014 at 19:15
-
$\begingroup$ I've added a little to the end of my answer $\endgroup$– Glen_bCommented Feb 26, 2014 at 22:37
$\begingroup$
$\endgroup$
5
You nay want to try conducting a Levene's test, this checks the assumption of homogeneous variance. If the p value is $< .25$, you assume unequal variances, and your safest bet is to run welch related tests. If your p value is $>.25$, run ANOVA, which for two groups is just a t-test. The assumption of normality is not a big deal when your sample size is so large.
Note that the p value of .25 for the Levene's test is a bit arbitrary, other books/professors may suggest a less conservative value such as .15 or .10, etc...
-
2$\begingroup$ W/ the reported N's, there is no point in running Levene's test. It will be significant. The larger point, however, is that the reported difference in variances will have little impact on the results. The question is Welch's t-test vs Mann-Whitney U-test, not Welch's t-test vs regular t-test. $\endgroup$ Commented Feb 25, 2014 at 1:53
-
$\begingroup$ @gung Fair enough, I am assuming that the Welch t-test would do given that the question asked for a comparison of the means. I believe Mann-Whitney U-test compares medians, but I am not certain. I guess I would not be able to look at the sample sizes and know that the Levene's test will be significant. The only only quick hand way I know of to check homogeneity of variances is if $\dfrac{\sigma_1}{\sigma_2}<4$, then the assumption holds. $\endgroup$– k6adamsCommented Feb 25, 2014 at 2:24
-
$\begingroup$ That's a standard (& reasonable) rule of thumb. (Technically, it's $\frac{\sigma_1^2}{\sigma_2^2}<4$, though.) Since that holds here, we needn't be too concerned, especially since the small group is 881. $\endgroup$ Commented Feb 25, 2014 at 2:28
-
1
-
$\begingroup$ Nevermind, it won't let me edit the comment, I will leave it up there to give the conversation context. $\endgroup$– k6adamsCommented Feb 25, 2014 at 2:31