# Does testing for assumptions affect type I error?

I just performed simple simulation. Made two "populations" with different means and the same variance. Since I prepared them I know that they: are normal, differs in location and both have the same scale. Then I wrote simple script in R which 1000 times draws samples from the both populations, test them for normality (Shapiro-Wilk), for equal variances (F) and for difference (t). If the assumptions of t-test are met I run t-test.

Well, in fact I run all of them one by one, but combine the results with logical AND operator. So, if IsNormal(A) AND IsNormal(B) AND VarsAreEqual(A, B) AND MeansAreDifferent(A, B), then I count the result. Finally I divide number of TRUE values and divide it by the number of iterations. So, every test of assumption may fail. Shapiro-Wilk may falsely (they ARE normal) reject H0 as well as F test (they HAVE equal variances). And I don't get values near 0.05!

So, my question is: is this an example of the "multiple testing" phenomena (like multiple comparisons)? If so, what about testing for assumptions? Such testing is crucial as if the assumptions are not met, the quantiles of sampling distribution of a given test may be completely different than expected and so given results may be completely useless. But testing for the assumptions clearly changes the alpha. Am I correct?

Perhaps we should use corrections for alpha, like Bonferroni?

data<- data.frame(populA=rnorm(1000, mean=1), populB=rnorm(1000, mean=3))

iternum <- 1000
alpha   <- 0.05
nsample <- 50
results <- array(FALSE, iternum);

for(i in 1:iternum) {
sampA <- sample(data$populA, 30); sampB <- sample(data$populB, 30);

result_t <- t.test(sampA, sampB)$p.value; result_F <- var.test(sampA, sampB)$p.value;
result_normA <- shapiro.test(sampA)$p.value; result_normB <- shapiro.test(sampB)$p.value;

results[i] <- (result_normA >= alpha) & (result_normB >= alpha) &
(result_F >= alpha) & (result_t < alpha)
}
> 1-(mean(results))

[1] 0.158

• Oh, I was too fast :) Of course, I know, that p.value of t-test WILL be near 0 as they really differ, so this doesn't affects my results. My questions concerns the tests that may fail - Shapiro and F. Commented Jun 2, 2014 at 22:19

Generally speaking, the answer is yes, both type I and type II error rates are impacted by choosing tests on the basis of tests of assumptions.

This is pretty well established with testing of equality of variance (for which several papers point it out), and testing normality. It should be expected that it will be the case in general.

The advice is usually along the lines of "if you can't make the assumption without testing, better to simply act as if the assumption doesn't hold".

So, for example, if you're trying to decide between the equal-variance and Welch-type t-tests, by default use the Welch test (though under equal sample size it is robust to violations of that assumption).

Similarly, in moderately-small$^*$ samples, you may be better off using a permutation test for location by default than testing for normality and then using a t-test if you fail to reject (in large samples, the t-test is usually level-robust enough that it's not likely to be that big an issue in most cases, if the sample is also large enough that you're not concerned about impact on power). Alternatively, the Wilcoxon-Mann-Whitney has very good power compared to the t-test at the normal, and would often be a very viable alternative.

[If for some reason you need to test it would be best to be aware of the extent to which the significance level and power of the tests may be affected under either arm of any resulting choice the test of assumptions leads you to. This will depend on the particular circumstances; for example simulation can be used to help investigate the behavior in similar situations.] * (but not very small, since the discreteness of the test statistic will limit the available significance levels too much; specifically, at very small sample sizes the smallest possible significance level may be impractically large)

A reference (with a link to more) on testing heteroskedasticity when choosing between equal-variance-t vs Welch-t location tests is here.

I also have one for the case of testing normality before choosing between the t test and the Wilcoxon-Mann-Whitney test (ref [3] here).

• Thank you for your answer, Glen_b. It hit me after the course of statistics I participated in, where teacher told us: "always! test for compliance with assumptions". I agree with you, there are some points I can follow your advice, for example Welch test you've mentioned. Moreover I know that if the distribution is symmetric (not necessarily normal, it can be even uniform or has many modes), the CLT works for us very well and it's not neccesary to test for normality. But what about other assumptions - many of tests have them. I cannot just assume that I will always use nonparametric ones or Commented Jun 2, 2014 at 23:16
• use Monte Carlo (bootstrapping or permutation) methods. Parametric regression, parametric ANOVA, parametric correlation and so on. Yes, indeed they are resistant to deviation from normality (CLT), but only for kurtosis and not for skewness, especially when there are "unbalanced" designs. Then... Should I always choose for nonparametrical or MC ones after I look at the scatterplot/boxplot/qq-plot? This is so "informal" that many of teachers will fail me during the exam calling it "reading tea leaves" :) Commented Jun 2, 2014 at 23:20
• But even Wilcoxon is not a "pure" nonparametric test as it has underlying assumption about equality of scale parameters (tested with Ansari Bradley test) :( I'm totally lost. It seems that the main diagnostic tool is scatterplot, boxplot and histogram (or kernel density curve) with Q-Q plot. But what about formal side of testing for assumptions? There are other assumptions, for example about sphericity of covariance matrix, autocovariance, stationarity in time series analysis... Commented Jun 2, 2014 at 23:28
• That's a lot of questions, I hardly know where to start. Commented Jun 2, 2014 at 23:36
• Let's write the questions here: 1. as we already know such testing changes errors, how to agree this with advices of well skilled teachers, often PhD? They say clearly: "test it formally, plots and diagrams are only additional tools". 2. Avoid formal testing? Yes, somethimes CLT will do the job for us as well as bootstrapping, but not always (there are other types of assumptions). 3. Always choose the nonparametric tests? But even some of them are "semi-non-parametric" and somethimes it involves lack of power (not every test has such a good properties as Wilcoxon (power ~ 80% of t-Test). Commented Jun 2, 2014 at 23:41

Just a thought on this topic.

It's certainly true that when testing for assumptions with many different tests, you are going to end up with a higher than $\alpha$ type I error rates (where $\alpha$ is the significance level test for each individual test), just by standard multiple testing issues.

But when testing assumptions, often type I error is not as bad as a type II error! For example, if your data is not normal and you falsely reject, then you may proceed to use a more robust test, which is (likely) still valid if the data is normal. On the other hand, if you fail to reject but the data is not normal, then you falsely assume normality, leading to the known problems of invalid assumptions.

When doing multiple comparison procedures (i.e. Bonferroni, etc.), we are trying to preserve the type I error rate, at the cost of higher type II error rates. In regards to testing assumptions, that seems like a silly notion, given the higher costs of type II errors.

You can, of course, take this one step further and ask why we are using significance levels for tests of assumptions to start with, when it's really power we should be concerned about. If you did, I would agree with you, but I'm not ready to open that whole can of worms at the moment.