To what degree is FWE correction required for multiple permutation based t-tests

Question

I am performing multiple t-tests to compare conditions of data. I have had to resort to permutation based t-tests (based on difference of means) due to high variance between samples.

My question is if I am using permutation based t-tests is FWE correction still required?

My understanding is that because is it non-parametric the probability of a false negative is not contingent on the number of tests applied to the same set of data. So FWE correction should not be required?

---

I will expand on my specific issue.

I am comparing scan-path data for subjects looking at two different types of faces (inverted and upright). I need to compare a number of measures for different face areas (eyes, nose , mouth, ears and 'other') between the areas. For this I have set up a t-test for each area, for each measure, and then correct for multiple comparisons. For example, I am testing the number of fixations, and fixation duration for each face area. I am constrained these tests in order to be able to compare to previous work. I have not been able to rely on any omnibus test thus far because of the non normality of my data greatly reduced the power of any omnibus test I have tried.

Answer 1

Expanding a bit on @MichaelLew 's answer (which is a good answer): Whether you have to correct for multiple comparisons, and whether family wise error is the best way to do this, have nothing to do with whether a test is parametric or semi-parametric or non-parametric. It has to do with philosophy.

Andrew Gelman and his colleagues wrote an excellent paper entitled "Why we (usually) don't have to worry about multiple comparisons". They mostly take a Bayesian approach to this problem (not surprisingly, as Dr. Gelman is one of the leading Bayesians) but I think it is true even in the classical or frequentist framework.

In addition to one point made early in Gelman's paper (the null hypothesis is rarely, if ever, strictly true), there are points raised by Jacob Cohen in his book on multiple regression (at least, in an earlier edition, I do not know if this is in the later editions), that it's a good question over what period or set of tests we should adjust - that is what is a "family" in the family wise error? He also says it is a subject on which reasonable people can differ.

Finally, although it may be obvious, anything that makes type I error smaller makes type II error larger. Sometimes type I errors are more serious than type II. But not always. Using a fixed p = .05 and power = .80 enshrines the idea that type II errors are 4 times worse than type I errors.

Answer 2

The family-wise error rate is not affected by whether the test is parametric or not. It is a product of the alpha level chosen for the hypothesis tests and the number of tests used and the nature of the decisions made on the basis of the test results.

You don't describe your data or your problem, so I can't give specific advice, but do you actually need multiple comparisons? How many hypotheses are you testing? Do you need to use a Neyman-Pearson approach of automatic decision rule-based inductive behaviour? Perhaps you could characterise any differences and make inference on the basis of their magnitudes and the scientific consideration of what they are and how they were observed.

I think that some permutations tests (those that are not based on ranks) have higher power when the variances of the groups are similar, so you might need to transform the data before analysis for best results.