I have close to a million data sets and whenever I run mean comparison test, either ANOVA or a t-test, I get a significance level of less than .0001 on SPSS. I'm concerned that my sample is so large that of course when I compare the means it will show up as significantly different. Can a sample be too large for ANOVA or a t-test?
No, a sample cannot be too large for an ANOVA or a t-test. You will almost invariably get statistically significant results because you have a great deal of power; however, this does not mean that you detect differences that are false. Indeed, regardless of how many cases you have, an effect that does not exist will not become significant. This is a common misconception.
A lot of power means that you might detect differences that are almost meaningless in terms of size, however. For example, maybe you find that two races are on average of different heights, but the difference is only half a millimetre.
Make sure to interpret the effect size associated with your statistical test. In this case, the p value is worth less than the effect size (as it often is)!
I suggest you look at the following (all very readable and non-technical):
- Anderson DR, Burnham KP, Thompson WL (2000) Null hypothesis testing: Problems, prevalence, and an alternative. Journal of Wildlife Management 64: 912-923.
- Gigerenzer G (2004) Mindless statistics. Journal of Socio-Economics 33: 587-606.
- Johnson DH (1999) The Insignificance of Statistical Significance Testing. The Journal of Wildlife Management 63: 763-772.