# Validity of R pwr package for extremely small sample sizes (<10)

Edit: this question is not:

• About post-hoc power analysis (at least not with the intent to use sample statistics in post-hoc analysis - I have pretty good population estimates and I want to show people to USE those)

• proper use of a common tool used extensively in the scientific community.
• Checking assumptions and approximations, including tacit ones, in this (or any other) tool.

As such, I believe closing this question (without even providing any hint of where and how to get it answered) actually hurts this community, by communicating the message that these checks are superfluous, while they really should be at the top of the list of priorities.

I hope this plea will get this reopened.

——————————-

Is the pwr package implemented with the small sample approximations (e.g. degrees of freedom in 1-sample t-test) described in Cohen 1988 or not? I can’t seem to find this in the docs.

If so, is there be a better choice for very small sample size power calculations?

———————

Optional reading: motivation for my question:

At work (manufacturing operation), we find it difficult to reproduce test results. To me it is clear that this is because we run severely underpowered studies. And my boss agreed that I should figure some stuff out.

I want to calculate power for some typical experimental “designs” we use.

The pwr package in R looks very convenient. The vignette references the book of Cohen: Statistical Power Analysis for the Behavioral Sciences (1988). The different field of application should have the same math, but I decided to check the book out anyway.

I notice some approximations in the book, which are valid when “small sample size” means > 30. For me, “very large sample size” sometimes means 10. I noticed the approximation now for 1-sample t-test and for paired t-test: the book mentions that it does not have tables with the correct degrees of freedom. Which leads to a really small difference for n > 30, but I suspect the approximation is probably not valid for n < 10. And I would like to check. Hence my question.

———————-

Edit: one of the comments asked for quotes, which is reasonable. I quote from Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.), as referenced in the pwr package.

here are two examples where Cohen mentions the use of an approximation: A not so severe example in footnote 1 on page 42 on t-test with unequal sample sizes: “This is because the table is treating the t test for n as based on df = 2n' - 2, when there are actually df=nA +nB - 2, a larger value.”

Page 46, on one-sample t-test has a more severe example: “The power tables were computed on the basis that n is the size of each of two samples and that therefore the t test would be based on 2(n- 1) degrees of freedom. In the one-sample case, t is perforce based on only n - 1 degrees of freedom.” And: “unless the sample size is small (say less than 25 or 30), the effect of the underestimation of the degrees of freedom is negligible.”

Note that I agree with this analysis and approximation. In my situation, however, I may be led astray, since I violate one of the conditions. Changing something that’s wrong into something that’s also wrong is not an improvement, so I would like to avoid that.
With modern computers, these approximations are no longer needed - but I could not to find in the docs whether the the R package is written with or without the approximation.

I know what we should do, but getting sample sizes of >20 will be extremely rare for us. Then again, we cán modify our experiments to get Effect Sizes that are larger than what Cohen considers as “large”. I just need to explain/motivate to people that we should do that. It WILL take effort to reach that point.

• I don't know the answer. With sample sizes that small, clearly power will be low, but I don't know how to get exact numbers. Some kind of resampling might be needed. However, there's an old book you might want to look at: Rick Hoyle, Statistical Strategies for Small Sample Research, from way back in 1999, I read it a long time ago and don't remember details, but I do remember it being useful. Commented Mar 3 at 10:51
• My comment does not answer your question, but maybe you're asking the wrong question to solve your problem. I may be mistaken, but it sounds a bit like you want to conduct post-hoc power calculation, which isn't really useful if it is what you intend to do. Is there a reason why you don't conduct power calculation with the minimal effect you want to be able to detect, for a level of power you find relevant (e.g. 0.8)? It will give you a relevant sample size for future tests. Commented Mar 3 at 12:48
• @Dave I guess OP is talking about Jacob Cohen's book on power analysis: Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. ISBN 978-0805802832 . There's a copy of it in pdf format, hosted on the University of Toronto website, but I'm not sure it is legal relative to the publisher's copyright so I don't give a direct link. There's a chapter about t-test, but the last chapter also gives further computational details about every test mentioned in the book. Commented Mar 3 at 15:54
• For very small sample sizes I often recommend against statistical tests in favor of providing confidence intervals for the effects of interest. CIs provide honest representations of what you know and what you don't. Unlike large P-values. Commented Mar 3 at 20:40
• Can you point more specifically to the place in the book where it "mentions that it does not have tables with the correct degrees of freedom" (by section or page number)? Unlike Cohen's book, the pwr package does not use tabulations internally; instead, it computes the relevant powers/sample sizes exactly (give or take tiny floating-point accuracy issues). My strong guess is that once you accept the approximations inherent in making the test assumptions (Normal distribution, independence, etc.), pwr gives you answers that are as exact as possible. Commented Mar 3 at 21:19

The advantage of open-source software is that when the manual's vague or ambiguous, you can check the code (the disadvantage is that you often have to). In R just inputting the name of a function returns its source code.

So for a two-tailed (tside=2), single-sample (tsample = 1), unpaired t-test; at significance level sig.level, with sample size n, the power to detect effects of size d is given by:—

nu <- (n - 1) * tsample
qu <- qt(sig.level/tside, nu, lower = FALSE)
pt(qu, nu, ncp = sqrt(n/tsample) * d, lower = FALSE) +
pt(-qu, nu, ncp = sqrt(n/tsample) * d, lower = TRUE)


qt & pt are the respective quantile & cumulative distribution functions for the t-distribution provided in the stats package. As you can see, they're supplied with the correct no. degrees of freedom nu.

It's worth noting that the probability of rejecting the null hypothesis because of a negative observed t-statistic is included in the total probability of rejecting the null hypothesis for a specified positive effect. This is negligible in high-powered experiments; but, as you suggest the ones you're interested in aren't, you may want to separate out the chance of inferring an effect in the wrong direction.

When estimation is an aim of the experiment, it can be useful to examine the distribution of lengths for a confidence interval about the mean. For large samples it's usually enough to say the length will be roughly a postulated standard error $$\frac{\sigma}{n}$$ multiplied by the difference between appropriate quantiles of the Gaussian distribution; but for small samples the lengths will be greater on average, & considerably more variable. For a given confidence level $$1-\alpha$$, the length of the interval is the standard error estimate multiplied by the difference between the corresponding quantiles of Student's t-distribution: \begin{align} L &= \frac{S}{\sqrt n}\cdot\left[F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1) -F^{-1}_\mathrm{t}(\tfrac{\alpha}{2};n-1)\right]\\ &= \frac{S}{\sqrt n}\cdot2F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1)\\ \end{align} As the sample variance follows a scaled chi-square distribution $$\frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1}$$ the lengths follow a scaled chi distribution. Let

$$k= \frac{\sqrt{n(n-1)}}{\left[2\sigma \cdot F^{-1}_\mathrm{t}(1-\tfrac{\alpha}{2};n-1)\right]^2}$$

& then the density & distribution functions for length can be conveniently expressed in terms of the chi-square density & distribution functions: \begin{align} f_L(l)&=2k^2 l f_{\chi^2}(k^2l^2; n-1)\\ F_L(l)&=F_{\chi^2}(k^2l^2; n-1) \end{align}

The mean length is given by

$$\operatorname{E} L = \frac{\sqrt 2}{k}\cdot\frac{\Gamma(\tfrac{n}{2})}{\Gamma(\tfrac{n-1}{2})}$$

'Underpowered' in this context would correspond to a good chance of obtaining an interval so wide that it would be liable to include importantly different values for the mean—a notion often easier for people to get their heads round.

I don't propose to go through every function in the pwr package, but e.g. the power analysis for tests comparing proportions depend on the Gaussian approximation to the binomial distribution, & would not therefore be all that accurate for for experiments with very small sample sizes. Simulation would be a better approach in this case.

It's somewhat tangential, but I'd question the applicability of the 'big', 'middle-sized', & 'little' effect size classification to manufacturing. If you increase the precision of a measurement system, say, why should a 'little' effect become 'big'? Effect sizes of practical significance are better stipulated according to engineering or financial criteria.

† Well, sometimes: see https://stackoverflow.com/q/19226816/1864816

• Thanks, I did not know of this feature of R! That answers a LOT of questions regarding implementation details!
– W_vH
Commented Mar 4 at 13:23
• On the Tangential: I like the description in effect size, because it makes it clear that “large” or “small” is best defined relative to number of standard deviations (at least for t-tests). This is somehow not intuitively clear to most of my colleagues… The designations for “large” etc that Cohen uses for behavioral sciences, clearly do not apply to manufacturing.
– W_vH
Commented Mar 4 at 14:03

#### Power Analysis and Potential Issues with Replication

There is some background information here that may be useful to determine why your results were not replicable. Some potential culprits:

• Did you determine results purely by $$p$$ values? Effect sizes, confidence intervals, and visualization will really go a long way to determining if the $$p$$ value is of any worth (I would argue, like Frank, that they are often meaningless at low sample sizes unless you have a lot of repeated measures...even then you are making a sacrifice to get there).
• What is the reliability of your measurements? Do they systematically differ between observations? This may also play a large part if it is not determined.
• Are the measurements standardized? Perhaps there are differences in measurement bore purely by the person measuring the data.
• Any potential ecological factors at play which influence the results?

With respect to statistical power, I don't know if you can really get any meaningful power of anything with $$n = 10$$ or lower unless you have a crazy effect size that makes any statistically significant effect obvious. As such, some options may be found below.

#### Small Sample Solutions

There is a nice book on small sample techniques that you may find useful called Small Sample Size Solutions: A Guide for Applied Researchers and Practitioners. It covers the following main themes:

• Bayesian methods for small samples (basically adding very informative priors to a small sample estimation). Keep in mind that your analysis would be heavily weighted by these priors.
• Repeated measures designs (RMANOVA, multilevel modeling, etc) such as Peter noted. This can make a huge difference sometimes. Crossed random effects models, for example, simply require multiplying two random effects to get more observations (e.g. $$10$$ people x $$200$$ trials = $$2000$$ observations).
• N-of-1 designs, which attempt to combine repeated measures and causality into one framework with very small samples. This might actually be your best option given the extremely low samples you are working with here.

Beyond all of that, I think numbers can only say so much. Doing some detective work on what's going on will go a long way with respect to the replication problem.

• An effect size that would be 'crazy' in the behavioural sciences might be quite sane in manufacturing. Generally quite a lot of work goes on in a factory to ensure the consistency of production processes & the precision of measurement systems; & it's not unusual for small, well-designed experiments to give clear results. Commented Mar 4 at 12:22
• That is true. Sometimes I forget how context-specific that can be. Commented Mar 4 at 12:37