I have read that "When you get more and more data, you can find statistically significant differences wherever you look"
My question is:
I think this comes from the fact that in the real world you don't really expect the standard null hypothesis to be true. If you're comparing the means of two populations, the null hypothesis says that $\mu_1 = \mu_2$, that is the two means are exactly equal. In many situations however a more accurate null hypothesis would say that $\mu_1$ and $\mu_2$ are almost equal (whatever that means).
For small sample sizes, the difference between means will only give a low p-value if the measured difference is "relatively" large. However for sufficiently large sample sizes even a tiny difference in means can become statistically significant, even though for practical purposes the numbers are the same.
There is some good information for this question here as well:
I have read that "When you get more and more data, you can find statistically significant differences wherever you look"
This is not always true, however if your null hypothesis is that two groups of people are exactly 100% the same then it is, because that null hypothesis will almost always or always be false. Instead if your null hypothesis is that the speed of light is 299,792,458 m/s and you measure this many times without using tools that are biased to make measurement error in one direction or the other, then you are not more likely to get significance.
Why is this case? (any intuitive examples that show this behavior?)
When it is the case, it is because the null hypothesis is false or there is some bias of the measuring tool.
Why do such increases in statistical difference do not necessarily imply that the observed effects are meaningful / important?
Because very small differences are just as likely to arise from other reasons other than the one you did the experiment to test (e.g. problem with the measuring device, baseline difference between groups) and there is no way to guess which has occurred. Note that this is always the case even if the effect is large, it is just less likely (as far as I can tell this is "hand-wavy", but intuitively obvious) to observe a large effect if all factors except your independent variable were held relatively constant.
Also very small differences usually do not provide any reason to take action based on the result. The cost of performing the action will usually outweigh the benefits.
Edit: Another thing is that obviously in the case of a null hypothesis predicted by theory, a non-significant result is important as your theory has been corroborated. Even in the case of the more common "always false" null hypothesis, data results in "non-significance" could be meaningful. Lack of significance, especially for large sample size, tells you that any effect/difference is small relative to background noise. I would say the practice of ignoring non-significant results is seriously flawed.
I also wish to highlight that you will not always find statistically significant results, even with nearly infinite data. Statistically significant results only represent what are likely to be true differences, regardless of size. If this difference does not exist, then the number of cases does not matter. Consider two samples of 10 million trees that are, on average, the exact same height. Having an overall sample of 20 million trees will never result in the difference being statistically significant. It is always important to assess the size of the effect when results are statistically significant. The results are important/meaningful within the context of what you are exploring. The importance will always depend. 1% difference may be very unimportant when considering shoe size, but be very meaningful when it represents odds of dying from a disease in a population of 10 billion.
It is not necessarily the case that you will always find significant differences as you increase sample size, but it becomes more and more likely. As several people have pointed out, truly identical samples may not result in a significant difference. What it does do is make very, very small differences much more likely to be detected - differences that we, in the real world, can't really act on in any meaningful fashion.
For example, if I told you that the average IQ of one group was 100.0001 and the other group 100.0002, would you really be able to treat the second group as "smarter" (given all the caveats around IQ as a measure of intelligence)?
I'll use an example from my own work: I was simulating an intervention in a hospital to help prevent patients from developing a particular disease. My data set was a number of simulated hospitals with treatment, and a number of hospitals without treatment.
The difference between them was statistically significant, and strongly so. This was entirely because the "No Treatment" hospitals had a few examples with slightly more infections. But in most meaningful ways, the two arms were identical. They had the same median number of cases, the same minimum, the same 75th percentile, and 95th percentile and even 99th percentile number of cases. The significance was entirely driven by a few edge cases at the extreme end of the distribution…and a large sample size.
The effect of the treatment was, in the real world, utterly undetectable and meaningless. But because I had a large sample size, it was statistically significant. If I had wanted it to be more so, I could have gone to dinner and let the simulation run longer, but that wouldn't have made the intervention any more effective.
Assume you have a treatment for the common cold which may or may not work. You administer it to one person, and that person gets better. It could be that your treatment is working, or it could be that the person just happened to get better by chance.
Now if you apply this treatment to two people and they both get better, this is already more convincing... what are the odds that two people you gave a treatment to both got better?
Now imagine that you give the treatment to a group of 500 people and they all get better, while in another group of 500 people that don't receive your treatment, only 10 get better. It could be that the group that you treated just happened to be more lucky, but as the number of people increases, the odds of that fluke happening become extremely small... it's more likely that your treatment actually has an effect.
The more data you have, the less likely it is that the patterns you observe are a fluke.
I believe user27840 has the right answer, but doesn't quite nail the intuition...
Let's take a common case: you are comparing the means of two groups, and your null hypothesis is that they are (exactly) equal. The test is also making an assumption about the distribution of the data, often that the data has a "normal distribution". (Technically, this phrase is wrong, but it's commonly used.)
The mean, by itself, doesn't have a lot of meaning. There is also the standard error of the mean, which reflects your uncertainty of the actual mean value. The standard error of the mean is tied in to the assumed distribution, and the more points you have in your calculation, the smaller it will be: the more certain you are of your calculation of the mean.
This is what works against you. With a small amount of data, the standard errors of your means will be larger, and unless the means are far apart, you will have to reject the null hypothesis (that they are equal) because the means may appear different but your uncertainty is large enough that you can't be sure. As you get more and more data, you become more and more sure of the mean -- the standard errors of your means shrink -- and the means can be closer and closer together but you will still be sure they're not the same.
The problem is, you are able to be certain about smaller and smaller diffences, but practically-speaking very small differences don't matter. Of course, the units you're measuring in and the subject matter determine what is a "very small" difference.
The traditional hypothesis testing framework seeks to keep type I error constant. The likelihood paradigm could be said to be more meaningful in this regard, as both type I and type II errors $\rightarrow 0$ as $n \rightarrow \infty$. This makes pure likelihood-based (as opposed to sample-space-based frequentist methods) approach less likely to find trivial differences.
I think this happens because people analyse their data after it has been observed. Further this analysis is not done "just for the sake of it" - you may change your mind about what is important based on this analysis (as you should).
As a simple example, the comparison of the means of two groups - say jurisdiction A has higher test scores than jurisdiction B. But after analysing the data, you find that the distribution of scores jurisdiction A has three modes, and jurisdiction B has two modes. After seeing this, why would you care if "overall" the means are different or not?
You are likely to dismiss the original hypothesis as "meaningless", and report the "interesting finding" of a multi modal distribution, possibly a statistical significant test to go with it. Follow up analysis would likely look for a variable that captures these modes.
This has been referred to as "researcher degrees of freedom" and does not get accounted for in your standard p-value. This is because your test statistic is now a function of your analysis. To see this, note that if you were to repeat the process (say in a follow up survey) you would analyse the new data set.
Additionally, this problem becomes worse as your data sets become larger, because there is much richer types of analysis you can do and more "real" differences that you can detect. For example, you can't detect three modes with a small data set.
