Why does Type I error always occur in a NHST? Or, is NHST too weak to tell the truth?

Question

Recently I have noticed that Type I error of a test is too much persistent. Even if all the values are almost equal (variance is extremely low) and tested to that value (true mean), still there is nearly 5% Type I error of the test. I am frightened and curious to rethink, are the people who have banned (or skeptic about) NHST (null hypothesis significance testing) truly correct?
Let's have a look at the following r code:

n=10000
t1err=0
set.seed(1235)
for (i in 1:n){
   x=rnorm(100, 10, 20)
   if (((t.test(x, mu=10))$p.value)<=0.05) (t1err=t1err+1) 
}
cat("Type I error rate in percentage is", (t1err/n)*100,"%")

It will produce roughly 5% as Type I error. Seems ok. But let's see this:

  n=10000
  t1err=0
  set.seed(1235)
  for (i in 1:n){
     x=rnorm(100, 10, 0.000001)
     if (((t.test(x, mu=10))$p.value)<=0.05) (t1err=t1err+1) 
  }
  cat("Type I error rate in percentage is", (t1err/n)*100,"%")

Each values are extremely close to 10. Still it produces about 5% Type I error. What conclusion we may draw from it? Or, am I missing something?

Answer 1

There is a lot of skepticism about the statistical tests, but the "problem" you point to is not one of the causes. You chose a significance level of 5%, so by your own choice you accepted you will incorrectly reject a true null-hypothesis in 5% of the times you applied this procedure. It is actually reasuring that you continue to find a 5% type I error rate regardless of how well the data fit and how large the data is. You decided that the Type I error rate should be 5%, so it would be very bad if the t-test somehow decided otherwise.

If you want to then it is trivial to reduce the type I error rate, just set your siginificance level at 1% or any other value you like. However, there is a price: you are less likely to detect a deviation from the null-hypothesis when such a deviation exists. This balance between these two errors is a central part of statistical testing.

Answer 2

The t-test is designed to work in just that way.

When you choose a significance level (a type I error rate) with a point null hypothesis, you have chosen the probability of rejection when the null hypothesis is true.

In both your simulations, the null hypothesis was true, so the rejection rate should have been your significance level. If you wanted a different rejection rate, you can hardly blame the t-test for what you chose.

It doesn't matter whether the population standard deviation is large or small, the t statistic is standardized so that when the null hypothesis is true (and the assumptions hold), the test statistic always has the relevant t-distribution -- the absolute value of the numerator of the statistic tends to be larger when $\sigma$ is larger - it's proportional to $\sigma$ - but the denominator is also proportional to $\sigma$, and the distribution of the ratio is invariant to the value of $\sigma$.

For example, if you double all the data values, both numerator and denominator double, and the t-statistic is unchanged.

[There are ways in which NHST may be deservedly criticized, but I don't think that the test actually having the type I error rate we chose it to have is one of them.]

Answer 3

I am just rephrasing what Maarten already said:

It's the idea of "frequentism", the statistical philosophy underlying NHST, to control the frequency of type I error at a fixed level alpha, which is commonly fixed at 5%.

So, if you generate data from H0 (as you do), a 5% type I error rate regardless of the variance of the distribution is exactly what we would expect from a properly working t-test.

Note that the t.test function as implemented in R tests for a difference of the means, while adjusting for the variance of the distribution. If you would fix the variance in the test, things would change, but then you would not create the data from H0 any more in both cases.