Skip to main content
8 of 8
added 245 characters in body
Scortchi
  • 31.6k
  • 9
  • 102
  • 281

The advantage of open-source software is that when the manual's vague or ambiguous, you can check the code (the disadvantage is that you often have to). In R just inputting the name of a function returns its source code.

So for a two-tailed (tside=2), single-sample (tsample = 1), unpaired t-test; at significance level sig.level, with sample size n, the power to detect effects of size d is given by:—

nu <- (n - 1) * tsample
qu <- qt(sig.level/tside, nu, lower = FALSE)
pt(qu, nu, ncp = sqrt(n/tsample) * d, lower = FALSE) + 
pt(-qu, nu, ncp = sqrt(n/tsample) * d, lower = TRUE)

qt & pt are the respective quantile & cumulative distribution functions for the t-distribution provided in the stats package. As you can see, they're supplied with the correct no. degrees of freedom nu.

It's worth noting that the probability of rejecting the null hypothesis because of a negative observed t-statistic is included in the total probability of rejecting the null hypothesis for a specified positive effect. This is negligible in high-powered experiments; but, as you suggest the ones you're interested in aren't, you may want to separate out the chance of inferring an effect in the wrong direction.

When estimation is an aim of the experiment, it can be useful to examine the distribution of lengths for a confidence interval about the mean. For large samples it's usually enough to say the length will be roughly a postulated standard error $\frac{\sigma}{n}$ multiplied by the difference between appropriate quantiles of the Gaussian distribution; but for small samples the lengths will be greater on average, & considerably more variable. For a given confidence level $1-\alpha$, the length of the interval is the standard error estimate multiplied by the difference between the corresponding quantiles of Student's t-distribution: $$\begin{align} L &= \frac{S}{\sqrt n}\cdot\left[F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1) -F^{-1}_\mathrm{t}(\tfrac{\alpha}{2};n-1)\right]\\ &= \frac{S}{\sqrt n}\cdot2F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1)\\ \end{align}$$ As the sample variance follows a scaled chi-square distribution $$ \frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1} $$ the lengths follow a scaled chi distribution. Let

$$k= \frac{\sqrt{n(n-1)}}{\left[2\sigma \cdot F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1)\right]^2}$$

& then the density & distribution functions for length can be conveniently expressed in terms of the chi-square density & distribution functions: $$\begin{align} f_L(l)&=2k^2 l f_{\chi^2}(k^2l^2; n-1)\\ F_L(l)&=F_{\chi^2}(k^2l^2; n-1) \end{align}$$

The mean length is given by

$$ \operatorname{E} L = \frac{\sqrt 2}{k}\cdot\frac{\Gamma(\tfrac{n}{2})}{\Gamma(\tfrac{n-1}{2})} $$

'Underpowered' in this context would correspond to a good chance of obtaining an interval so wide that it would be liable to include importantly different values for the mean—a notion often easier for people to get their heads round.

I don't propose to go through every function in the pwr package, but e.g. the power analysis for tests comparing proportions depend on the Gaussian approximation to the binomial distribution, & would not therefore be all that accurate for for experiments with very small sample sizes. Simulation would be a better approach in this case.

It's somewhat tangential, but I'd question the applicability of the 'big', 'middle-sized', & 'little' effect size classification to manufacturing. If you increase the precision of a measurement system, say, why should a 'little' effect become 'big'? Effect sizes of practical significance are better stipulated according to engineering or financial criteria.


† Well, sometimes: see https://stackoverflow.com/q/19226816/1864816

Scortchi
  • 31.6k
  • 9
  • 102
  • 281