8
$\begingroup$

All other things being equal is there ever a time when a t-test would be preferable to a permutation test? Computational time may be one reason, but for our purposes assume it does not matter.

Isn't permutation test a superior option in this case?

$\endgroup$
8
  • $\begingroup$ Does this answer your question? Why would parametric statistics ever be preferred over nonparametric? $\endgroup$ Mar 17, 2020 at 15:47
  • 2
    $\begingroup$ @StephanKolassa Well, I don't think so. That question is focusing on typical nonparametric tests, not specifically the permutation test. For example, there are many situations where a corresponding nonparametric test is not available, e.g. for two-way mixed ANOVA interaction effect. So the answer to that question would be "No". But the permutation test could do the job, by simply permuting labels and calculate the extremeness of the statistic using real data. I am posting this question for the permutation test only. Besides, there was not a lot of discussion on permutation test there,. $\endgroup$
    – WCMC
    Mar 17, 2020 at 15:53
  • $\begingroup$ A permutation test just gives the p-value. The t-test provides more information - that's perhaps one reason. $\endgroup$ Mar 17, 2020 at 16:49
  • $\begingroup$ Hi @JeremyMiles, thanks. But what information does the t-test give more than the permutation test? $\endgroup$
    – WCMC
    Mar 17, 2020 at 16:52
  • 1
    $\begingroup$ I don't think it is necessary. The permutation test can use t-statistic, so the t-test does not give more information w.r.t t-statistic for the permutation test. Quesiton is, can we use permutation test (almost) always? $\endgroup$
    – WCMC
    Mar 17, 2020 at 18:59

1 Answer 1

6
$\begingroup$

I am a fan of permutation tests in general, but there are some important considerations to their construction that could be interpreted as disadvantages. They have benefits, too, though, that make me come down in favor of permutation tests in the vast majority of situations.

Benefits

  1. For designed experiments, you can always construct a permutation test that matches your experimental design. There are a number of papers about this, but one of my favorites is Permutation Tests for Multi-Factorial Analysis of Variance by Marti Anderson.

  2. When constructed properly, the permutation test does not really rely on unverifiable assumptions about your data. This is discussed at length by Tukey in Tightening the Clinical Trial which is worth a read if you are interested in non-parametric analyses.

These two benefits are worth a lot - the p-values (and confidence intervals) that are generated by a permutation test can justified with solely the construction of the test. Moreover, the main benefit to using an asymptotic test like the t-test is that it is much easier to compute, but asymptotic tests can never be more exact than the permutation test. Therefore, if computational power is not limiting, you should go for the permutation test.

That's not to say permutation tests are perfect, however.

Drawbacks

  1. Permutation tests are inefficient compared to asymptotic tests. When you have only a few observations, it might be impossible to control alpha at the level you want and still do a permutation test. This can be mitigated by collecting more data, however.

  2. Choice of test statistic is a much more important drawback to keep in mind. As a general rule of thumb, for any given metric $\beta$, you should use $\beta /s.e.(\beta)$ as the test statistic. Permutation tests tend to be sensitive to differences in distributions rather than differences in parameter, but using a pivotal (or approximately pivotal) test statistic fixes this problem. One handy way of doing that is just dividing your comparison of interest by an estimate of its standard error. This is discussed by Chung and Romano in Exact and asymptotically robust permutation tests. This does not completely fix the issue - comparing asymmetric distributions with massive heteroscedasticity is still difficult, but it's also quite difficult to do with asymptotic tests.

On the whole, I come down on the side of the permutation test for two main reasons: 1. you can analyze any experimental design with a permutation test (though not necessarily an exact one) and 2. its drawbacks can be mitigated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.