In literature, I normally see authors use a two sample permutation test on normal data to show that it works as well as the two sample t-test. However, the real power for permutation tests should be the non-parametric properties.

Does anyone have a good example of non-normal two sample data that fails the t-test but where the permutation test succeeds?

  • 1
    $\begingroup$ I suggest you reword slightly "the real power for permutation tests should be the non-parametric properties"; perhaps something like "the real advantage of permutation tests should lie in their ability to handle non-normal data" or similar? You might want to define more clearly what you mean by the data "failing" the t-test; do you mean that the t-test fails to detect a true difference? $\endgroup$ – Silverfish Sep 29 '15 at 19:47
  • $\begingroup$ (A lot of people use "succeed" to mean "gives a p-value below 0.05" and "fails" to mean "gives a non-significant result", but that is not a good way to think about hypothesis testing! It's not the job of a test to give you the answer that you want...) $\endgroup$ – Silverfish Sep 29 '15 at 21:52
  • $\begingroup$ Please define what you mean for a t-test to "fail". Away from the normal it won't have its nominal properties (e.g. significance level, power behavior), but what would have to happen for that to be 'failure'? $\endgroup$ – Glen_b Sep 30 '15 at 3:28

I think a good example is the space shuttle data as given in the textbook $\it{The \ Statistical \ Sleuth}.$ The number of O-ring incidents is given by launch temperature as

Temp $ \ \ \ \ \ \ \ \ \ \ \ \ $ Number of O-Ring Incidents

$\lt 65^{\circ} \ \ \ \ \ \ \ \ \ \ \ \ \ \ 1,1,1,3$

$\gt 65^{\circ} \ \ \ \ \ \ \ \ \ \ \ \ \ \ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2$

A Welch's t-test on this data gives a p-value of $0.0853$

The permutation test gives $0.0099$

  • 1
    $\begingroup$ Which one is "correct"? Is it even possible to tell, given that the threshold temperature was determined post hoc? $\endgroup$ – whuber Sep 29 '15 at 23:38
  • $\begingroup$ Neither is correct. All models are wrong - but some are useful. $\endgroup$ – soakley Sep 30 '15 at 14:04
  • 1
    $\begingroup$ I don't understand, then, how this example responds to the question. Could you perhaps be more explicit about what you understand a "failure" and a "success" of a test to be? $\endgroup$ – whuber Sep 30 '15 at 15:16
  • $\begingroup$ If a significance level of $0.05$ were used, the Welch t-test fails to reject, while the permutation test "succeeds" in rejecting and indicates differing means. $\endgroup$ – soakley Sep 30 '15 at 22:03
  • 1
    $\begingroup$ So what you are demonstrating is that these two tests do not always give identical results. I attribute the trivial nature of that result to the nature of the question, not to your answer. But absent a response from the OP to the requested clarification, it would be desirable to show that the tests can also go in the other direction: The Welch test can reject while the permutation test will fail to reject. $\endgroup$ – whuber Sep 30 '15 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.