Type II error or no error?

Question

I get stuck in the following question:

You set up a two-sided hypothesis test for a population mean μ with a null hypothesis of​: μ=100. You chose a significance level of α=0.05. The p-value calculated from the data is 0.12, and hence you failed to reject the null hypothesis. Suppose that after your analysis was completed and published, an expert informed you that the true value of μ is 104. How would you describe the result of your analysis?

I wonder if there might be no error? The true mean is around that in the null hypothesis, then I thought there would be no error. I mean if the true value of μ is 100.000001, the error would be the same as when it is 104. But there should be a buffer for the null hypothesis.

Any suggestions would be highly appreciated.

Answer 1

Suppose you used a 1-sample t test at level 5% to test $H_0: \mu = 100$ vs. $H_a: \mu \ne 100.$ You used a sample of size $n = 20$ without checking the power against various alternatives.

Now, somehow you know that the population from which you sampled is $\mathsf{Norm}(\mu=104,\sigma=15).$ So now you know you failed to reject $H_0,$ when it is false, a Type II Error.

What is the probability of rejection in these circumstances? That is, what is the power of the test against alternative value $\mu=104?$ One can get an exact value using the noncentral t distribution. In case the point of this question is for you to do that computation, I will get a close approximation to the power, only about $0.21,$ by simulation in R.

set.seed(804)
pv = replicate(10^5, t.test(rnorm(20, 104, 15), mu=100)$p.val)
mean(pv <=.05)
[1] 0.20612

So, it is hardly a surprise that you didn't reject. In fact, the probability of getting a P-value of $0.12$ or greater is about $0.65.$

mean(pv > .12)
[1] 0.65168

Here is a histogram of the distribution of the P-value. The power of the test in the circumstances described is approximately the area of the left-most bar in this histogram. (If $H_0$ were true, then the distribution of the P-value would be standard uniform.)

hist(pv, prob=T, col="skyblue2")
abline(v = .12, col="red")