Power analysis using R gives contradicting result using an estimated sd

Question

Suppose we are to find if the yield of a crop is greater than the previous yield $\mu_0$ with a specified level of significance $\alpha=0.05$. The population variance is unknown. My first sample has a size of 8, and has a sample mean of $\overline{x}$ and sample variance $\hat{\sigma}^2\approx0.085$. Then we use the fact that $T_{\mu_0}=\frac{\overline{\xi}-\mu_0}{\hat{\sigma}/\sqrt{n}}\sim t_{n-1}$ under $H_0$ to find the critical region for our test statistic $T_{\mu_0}$ as $(c,+\infty)$. Now if we want to find the power of our test (note that the sample size $n$ is fixed), we need to specify the estimated mean $\mu_*$, and use the fact that $T_{\mu_*}=\frac{\overline{\xi}-\mu_*}{\hat{\sigma}/\sqrt{n}}\sim t_{n-1}$ under this $H_1:\mu_\xi=\mu_*>\mu_0$ and calculate the probability $\mathbb{P}(T_{\mu_*}>c_*|\mu=\mu_*)$ where $c_*=c-\frac{\mu_*-\mu_0}{\hat{\sigma}/\sqrt{n}}$. It is known that $\delta=\mu_*-\mu_0=0.2$. Using the data above (the sample variance and delta), it can be calculated that the t-test on this problem with a sample size of 8 has a power of around $0.55$.

The problem is that, if we want to increase the sample size $n$ so that we could make our test have a power higher than a specified value $B$ (in this case suppose $B=0.8$), then we need to find a $\hat{\sigma}$ that is suitable for the entire population. If we choose $\hat{\sigma}\approx0.2$, then after we run the code

power.t.test(power=0.8,delta=0.2,sd=0.2,sig.level=0.05,type="one.sample",alternative="one.sided")

we get the calculated $n=8$, which is absurd. The same result occurs when I calculate it manually and assuming $\hat\sigma=0.2$.

In my opinion, this is because of the sd we use is $0.2<\sqrt{0.085}\approx0.289$ where 0.289 is the sample sd. How are we able to fix the test by choosing the correct sd? Or is there anything that I did wrong (e.g. I used the sample variance to find the amended critical value $c_*$, but should we use 0.2 instead)?

Answer 1

In order find $n$ for a given power (or power for a given $n)$ you need to specify $\alpha, \delta,$ and $\sigma.$ You choose $\alpha = .05 = 5\%,$ and $\delta = 0.2$ Typically, you will not know $\sigma$ and it is reasonable to estimate it by the sample standard deviation $S$ from a previous similar experiment. Of course, it will make a difference in your computations whether you use $\sigma = 0.2$ or the estimate $\sigma = 0.289.$

The formula connecting $\alpha,\delta,\sigma,n$ and power $\pi$ uses a noncentral t distribution. However, you can approximate the power for a one-sided test according to specifications $\alpha = 0.05, \delta = 0.2,$ $\sigma = 0.2,$ and $n = 8$ as shown below. I get power about $0.8.$ So it seems that $n = 8$ is not an 'absurd' answer.

set.seed(2021)
pv = replicate(10^5, t.test(rnorm(8, 0.2, 0.2), mu = 0, alt="g")$p.val)
mean(pv <= 0.05)
[1] 0.81548

If you increase the sample size to $n = 10$ in this simulation, the power increases to almost $0.9.$

set.seed(1111)
pv = replicate(10^5, t.test(rnorm(10, 0.2, 0.2), mu = 0, alt="g")$p.val)
mean(pv <= 0.05)
[1] 0.89657

Notes: (1) comments on R code: The numeric vector pv has P-values from $10^5.$ The logical vector pv <= 0.05 has $10^5$ TRUEs and FALSEs, and its mean is the proportion of its TRUEs (that is Rejections), which is the approximate power of the test. If you use $10^5$ iterations, power should be accurate to about two decimal places.

(2) I hope you can use the power.t.test procedure in R to verify the results of the simulations. Also, there are some online power and sample size "calculators" that are not difficult to use and that give reasonable answers.

(3) Finally, you can look at an advanced applied statistics book or a math stat book to find the formula that uses the non-central t distribution. In R, the first two parameters of the CDF pt of a t distribution are a value of t and the degrees of freedom; the third parameter is the non-centrality parameter. The formula shows that the ratio $\delta/\sigma$ is a key quantity.

Answer 2

Not really an answer since @BruceET provided already a detailed one, but more a comment on:

we get the calculated 𝑛=8, which is absurd.

As mentioned in @BruceET's answer in bullet (3), the effect size $\delta$ and the standard deviation $\sigma$ are key. A small effect size will be much more difficult to detect if there is high variability in the data (and therefore requiring bigger sample size $n$)... and vice-versa.

Below is a visualization of the required sample size $n$ depending on the ratio $\delta/\sigma$ (using power.t.test). As you will see, the requirement for sample size exponentially increases when we want to detect an effect that is smaller than standard deviation.