Validity of R pwr package for extremely small sample sizes (<10)

Question

Edit: this question is not:

• about programming
• About post-hoc power analysis (at least not with the intent to use sample statistics in post-hoc analysis - I have pretty good population estimates and I want to show people to USE those)

This question is about:

• proper use of a common tool used extensively in the scientific community.
• Checking assumptions and approximations, including tacit ones, in this (or any other) tool.

As such, I believe closing this question (without even providing any hint of where and how to get it answered) actually hurts this community, by communicating the message that these checks are superfluous, while they really should be at the top of the list of priorities.

I hope this plea will get this reopened.

——————————-

Is the pwr package implemented with the small sample approximations (e.g. degrees of freedom in 1-sample t-test) described in Cohen 1988 or not? I can’t seem to find this in the docs.

If so, is there be a better choice for very small sample size power calculations?

Thanks for your reply.

———————

Optional reading: motivation for my question:

At work (manufacturing operation), we find it difficult to reproduce test results. To me it is clear that this is because we run severely underpowered studies. And my boss agreed that I should figure some stuff out.

I want to calculate power for some typical experimental “designs” we use.

The pwr package in R looks very convenient. The vignette references the book of Cohen: Statistical Power Analysis for the Behavioral Sciences (1988). The different field of application should have the same math, but I decided to check the book out anyway.

I notice some approximations in the book, which are valid when “small sample size” means > 30. For me, “very large sample size” sometimes means 10. I noticed the approximation now for 1-sample t-test and for paired t-test: the book mentions that it does not have tables with the correct degrees of freedom. Which leads to a really small difference for n > 30, but I suspect the approximation is probably not valid for n < 10. And I would like to check. Hence my question.

———————-

Edit: one of the comments asked for quotes, which is reasonable. I quote from Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.), as referenced in the pwr package.

here are two examples where Cohen mentions the use of an approximation: A not so severe example in footnote 1 on page 42 on t-test with unequal sample sizes: “This is because the table is treating the t test for n as based on df = 2n' - 2, when there are actually df=nA +nB - 2, a larger value.”

Page 46, on one-sample t-test has a more severe example: “The power tables were computed on the basis that n is the size of each of two samples and that therefore the t test would be based on 2(n- 1) degrees of freedom. In the one-sample case, t is perforce based on only n - 1 degrees of freedom.” And: “unless the sample size is small (say less than 25 or 30), the effect of the underestimation of the degrees of freedom is negligible.”

Note that I agree with this analysis and approximation. In my situation, however, I may be led astray, since I violate one of the conditions. Changing something that’s wrong into something that’s also wrong is not an improvement, so I would like to avoid that.
With modern computers, these approximations are no longer needed - but I could not to find in the docs whether the the R package is written with or without the approximation.

I know what we should do, but getting sample sizes of >20 will be extremely rare for us. Then again, we cán modify our experiments to get Effect Sizes that are larger than what Cohen considers as “large”. I just need to explain/motivate to people that we should do that. It WILL take effort to reach that point.

Answer 1

The advantage of open-source software is that when the manual's vague or ambiguous, you can check the code (the disadvantage is that you often have to). In R just inputting the name of a function returns its source code.^†

So for a two-tailed (tside=2), single-sample (tsample = 1), unpaired t-test; at significance level sig.level, with sample size n, the power to detect effects of size d is given by:—

nu <- (n - 1) * tsample
qu <- qt(sig.level/tside, nu, lower = FALSE)
pt(qu, nu, ncp = sqrt(n/tsample) * d, lower = FALSE) + 
pt(-qu, nu, ncp = sqrt(n/tsample) * d, lower = TRUE)

qt & pt are the respective quantile & cumulative distribution functions for the t-distribution provided in the stats package. As you can see, they're supplied with the correct no. degrees of freedom nu.

It's worth noting that the probability of rejecting the null hypothesis because of a negative observed t-statistic is included in the total probability of rejecting the null hypothesis for a specified positive effect. This is negligible in high-powered experiments; but, as you suggest the ones you're interested in aren't, you may want to separate out the chance of inferring an effect in the wrong direction.

When estimation is an aim of the experiment, it can be useful to examine the distribution of lengths for a confidence interval about the mean. For large samples it's usually enough to say the length will be roughly a postulated standard error $\frac{\sigma}{n}$ multiplied by the difference between appropriate quantiles of the Gaussian distribution; but for small samples the lengths will be greater on average, & considerably more variable. For a given confidence level $1-\alpha$, the length of the interval is the standard error estimate multiplied by the difference between the corresponding quantiles of Student's t-distribution: $$\begin{align} L &= \frac{S}{\sqrt n}\cdot\left[F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1) -F^{-1}_\mathrm{t}(\tfrac{\alpha}{2};n-1)\right]\\ &= \frac{S}{\sqrt n}\cdot2F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1)\\ \end{align}$$ As the sample variance follows a scaled chi-square distribution $$ \frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1} $$ the lengths follow a scaled chi distribution. Let

$$k= \frac{\sqrt{n(n-1)}}{\left[2\sigma \cdot F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1)\right]^2}$$

& then the density & distribution functions for length can be conveniently expressed in terms of the chi-square density & distribution functions: $$\begin{align} f_L(l)&=2k^2 l f_{\chi^2}(k^2l^2; n-1)\\ F_L(l)&=F_{\chi^2}(k^2l^2; n-1) \end{align}$$

The mean length is given by

$$ \operatorname{E} L = \frac{\sqrt 2}{k}\cdot\frac{\Gamma(\tfrac{n}{2})}{\Gamma(\tfrac{n-1}{2})} $$

'Underpowered' in this context would correspond to a good chance of obtaining an interval so wide that it would be liable to include importantly different values for the mean—a notion often easier for people to get their heads round.

I don't propose to go through every function in the pwr package, but e.g. the power analysis for tests comparing proportions depend on the Gaussian approximation to the binomial distribution, & would not therefore be all that accurate for for experiments with very small sample sizes. Simulation would be a better approach in this case.

It's somewhat tangential, but I'd question the applicability of the 'big', 'middle-sized', & 'little' effect size classification to manufacturing. If you increase the precision of a measurement system, say, why should a 'little' effect become 'big'? Effect sizes of practical significance are better stipulated according to engineering or financial criteria.

---

† Well, sometimes: see https://stackoverflow.com/q/19226816/1864816

Answer 2

Power Analysis and Potential Issues with Replication

There is some background information here that may be useful to determine why your results were not replicable. Some potential culprits:

• Did you determine results purely by $p$ values? Effect sizes, confidence intervals, and visualization will really go a long way to determining if the $p$ value is of any worth (I would argue, like Frank, that they are often meaningless at low sample sizes unless you have a lot of repeated measures...even then you are making a sacrifice to get there).
• What is the reliability of your measurements? Do they systematically differ between observations? This may also play a large part if it is not determined.
• Are the measurements standardized? Perhaps there are differences in measurement bore purely by the person measuring the data.
• Any potential ecological factors at play which influence the results?

With respect to statistical power, I don't know if you can really get any meaningful power of anything with $n = 10$ or lower unless you have a crazy effect size that makes any statistically significant effect obvious. As such, some options may be found below.

Small Sample Solutions

There is a nice book on small sample techniques that you may find useful called Small Sample Size Solutions: A Guide for Applied Researchers and Practitioners. It covers the following main themes:

• Bayesian methods for small samples (basically adding very informative priors to a small sample estimation). Keep in mind that your analysis would be heavily weighted by these priors.
• Repeated measures designs (RMANOVA, multilevel modeling, etc) such as Peter noted. This can make a huge difference sometimes. Crossed random effects models, for example, simply require multiplying two random effects to get more observations (e.g. $10$ people x $200$ trials = $2000$ observations).
• N-of-1 designs, which attempt to combine repeated measures and causality into one framework with very small samples. This might actually be your best option given the extremely low samples you are working with here.

Beyond all of that, I think numbers can only say so much. Doing some detective work on what's going on will go a long way with respect to the replication problem.