I read an interesting statement: running the same data through multiple tests no longer counts as multiple comparisons.
I am fairly confident this is incorrect if you are testing different hypotheses e.g. testing for hundreds of different lags on the same data will yield some false significant results (if you have enough data to test for large lags).
However, if you are testing exactly the same hypothesis, but with two different tests, is this still a multiple comparison?
My thinking so far is that firstly it is rarely if ever true that two tests have exactly the same null hypothesis. e.g. the Shapiro-wilk test and Jarque–Bera test, while both ostensibly "normality tests" are testing slightly different null hypotheses, that the data came from a normal distribution and that the data cam from a distribution with skew and kurtosis equal to a normal distribution. Are there any tests with exactly the same null hypothesis?
If there are tests that against exactly the same null hypothesis, would this count as a multiple comparison? And if the tests have exactly the same null hypothesis, is it possible for them to yield different results given the same data?
