I have close to a million data sets and whenever I run mean comparison test, either ANOVA or a t-test, I get a significance level of less than .0001 on SPSS. I'm concerned that my sample is so large that of course when I compare the means it will show up as significantly different. Can a sample be too large for ANOVA or a t-test?
$\begingroup$
$\endgroup$
1
-
4$\begingroup$ The question is pretty well covered, so just a comment - it's more a question that you're asking a question you already know the answer to (you don't really think the population means will be exactly equal, do you?), and you have sufficient sample size to give the already-obvious-but-not-very-interesting-answer that there are in fact differences, even if they're tiny. Effect sizes and confidence intervals are more useful than p-values, and your interesting questions tend to go more to issues of bias and describing the differences that are there and judging their practical importance. $\endgroup$– Glen_bCommented Apr 12, 2013 at 0:51
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
No, a sample cannot be too large for an ANOVA or a t-test. You will almost invariably get statistically significant results because you have a great deal of power; however, this does not mean that you detect differences that are false. Indeed, regardless of how many cases you have, an effect that does not exist will not become significant. This is a common misconception.
A lot of power means that you might detect differences that are almost meaningless in terms of size, however. For example, maybe you find that two races are on average of different heights, but the difference is only half a millimetre.
Make sure to interpret the effect size associated with your statistical test. In this case, the p value is worth less than the effect size (as it often is)!
-
2$\begingroup$ Quite correct (+1). The only problem that I could see is if you run into memory and execution time problems. With modern computers, I think that would take a lot more than 1 million cases for simple things like ANOVA or t-test. $\endgroup$ Commented Apr 12, 2013 at 0:09
-
$\begingroup$ What happens when sample size is equal to population size? How does one pick p-value then? $\endgroup$– SharathCommented Nov 25, 2018 at 6:57
-
$\begingroup$ @Sharath That sounds like a new question that could be worth posting separately. $\endgroup$– DaveCommented Dec 7, 2022 at 22:10
$\begingroup$
$\endgroup$
I suggest you look at the following (all very readable and non-technical):
- Anderson DR, Burnham KP, Thompson WL (2000) Null hypothesis testing: Problems, prevalence, and an alternative. Journal of Wildlife Management 64: 912-923.
- Gigerenzer G (2004) Mindless statistics. Journal of Socio-Economics 33: 587-606.
- Johnson DH (1999) The Insignificance of Statistical Significance Testing. The Journal of Wildlife Management 63: 763-772.
