# What is an example of data where the permutation test succeeds but a normal t-test fails?

In literature, I normally see authors use a two sample permutation test on normal data to show that it works as well as the two sample t-test. However, the real power for permutation tests should be the non-parametric properties.

Does anyone have a good example of non-normal two sample data that fails the t-test but where the permutation test succeeds?

• I suggest you reword slightly "the real power for permutation tests should be the non-parametric properties"; perhaps something like "the real advantage of permutation tests should lie in their ability to handle non-normal data" or similar? You might want to define more clearly what you mean by the data "failing" the t-test; do you mean that the t-test fails to detect a true difference? – Silverfish Sep 29 '15 at 19:47
• (A lot of people use "succeed" to mean "gives a p-value below 0.05" and "fails" to mean "gives a non-significant result", but that is not a good way to think about hypothesis testing! It's not the job of a test to give you the answer that you want...) – Silverfish Sep 29 '15 at 21:52
• Please define what you mean for a t-test to "fail". Away from the normal it won't have its nominal properties (e.g. significance level, power behavior), but what would have to happen for that to be 'failure'? – Glen_b Sep 30 '15 at 3:28

I think a good example is the space shuttle data as given in the textbook $\it{The \ Statistical \ Sleuth}.$ The number of O-ring incidents is given by launch temperature as

Temp $\ \ \ \ \ \ \ \ \ \ \ \$ Number of O-Ring Incidents

$\lt 65^{\circ} \ \ \ \ \ \ \ \ \ \ \ \ \ \ 1,1,1,3$

$\gt 65^{\circ} \ \ \ \ \ \ \ \ \ \ \ \ \ \ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,2$

A Welch's t-test on this data gives a p-value of $0.0853$

The permutation test gives $0.0099$

• Which one is "correct"? Is it even possible to tell, given that the threshold temperature was determined post hoc? – whuber Sep 29 '15 at 23:38
• Neither is correct. All models are wrong - but some are useful. – soakley Sep 30 '15 at 14:04
• I don't understand, then, how this example responds to the question. Could you perhaps be more explicit about what you understand a "failure" and a "success" of a test to be? – whuber Sep 30 '15 at 15:16
• If a significance level of $0.05$ were used, the Welch t-test fails to reject, while the permutation test "succeeds" in rejecting and indicates differing means. – soakley Sep 30 '15 at 22:03
• So what you are demonstrating is that these two tests do not always give identical results. I attribute the trivial nature of that result to the nature of the question, not to your answer. But absent a response from the OP to the requested clarification, it would be desirable to show that the tests can also go in the other direction: The Welch test can reject while the permutation test will fail to reject. – whuber Sep 30 '15 at 22:56