1
$\begingroup$

I read an interesting statement: running the same data through multiple tests no longer counts as multiple comparisons.

I am fairly confident this is incorrect if you are testing different hypotheses e.g. testing for hundreds of different lags on the same data will yield some false significant results (if you have enough data to test for large lags).

However, if you are testing exactly the same hypothesis, but with two different tests, is this still a multiple comparison?

My thinking so far is that firstly it is rarely if ever true that two tests have exactly the same null hypothesis. e.g. the Shapiro-wilk test and Jarque–Bera test, while both ostensibly "normality tests" are testing slightly different null hypotheses, that the data came from a normal distribution and that the data cam from a distribution with skew and kurtosis equal to a normal distribution. Are there any tests with exactly the same null hypothesis?

If there are tests that against exactly the same null hypothesis, would this count as a multiple comparison? And if the tests have exactly the same null hypothesis, is it possible for them to yield different results given the same data?

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ See here for an example of two tests with the same null hypothesis but different rejection regions. $\endgroup$ – Scortchi - Reinstate Monica♦ Nov 3 '13 at 1:38
  • 1
    $\begingroup$ If the tests reject basically the same cases, then they're redundant - you're completely wasting your time doing a second test. If the tests don't generally reject the same cases, then you're effectively doing multiple testing, though the extent of it depends on how nearly independent they are - the more independent they are, the more it counts fully as multiple testing, while the more dependent they are, the more it's a complete waste of time to do it at all. If you want to test one hypothesis, pick the most powerful test you can against alternatives you care about and be done with it. $\endgroup$ – Glen_b -Reinstate Monica Nov 3 '13 at 2:54
  • $\begingroup$ To respond to the questions at the end - yes, you can test the same null with two test statistics that don't order the sample space quite the same way, as long as you accompany the tests with the right assumptions. I can use a Mann Whitney as a test of a difference in means for data assumed to be iid normal (apart from possible location-shift) by adding the assumption that the data are iid normal apart from location shift. It's just not the best possible way to test that if the assumption is exactly true. $\endgroup$ – Glen_b -Reinstate Monica Nov 3 '13 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.