Advice on how to answer the "how large of a sample size do we need to show results" question to non-quant people in a business setting

Not sure if this question is too vague, but I'll give it a shot.

I've received many questions from non-quantitative colleagues along the lines of "what sample size do we need to trust these results" or "what sample size is statistically significant" (I know this second one doesn't make sense). They are looking for one number and will inevitably trust results where n >= threshold, and distrust results where n < threshold. I don't think that it should be this black and white.

The first thing that comes to my mind is power calculations, but I don't think they're appropriate for situations where, say, we already collected the data and are calculating some statistic (could be a simple mean, could be a t-test, etc.).

I would prefer to run statistical tests, calculate means and CIs, etc. in all cases, UNLESS I think that some normality assumption is violated. In other words, the sample size might be too small to assume the test statistic follows an xyz distribution. But then again, business people love to run away with point estimates and ignore confidence intervals. So it's sort of an art, and I'm guessing some responses will center around using business sense.

Any open-ended responses or general words of wisdom would be greatly appreciated.

• In polling to get Yes/No data about usefulness of a new product or for agreement with a proposition at an upcoming election the margin of error $E$ for a confidence interval (CI) of the proportion $p$ of Yes's is about $1/\sqrt{n}.$ So in a poll of $n=2500$ randomly chosen subjects from the population of interest, $E$ for a 95% CI is about $\pm 2\%.$ And it takes about $n=1100$ for $E \approx 3\%.$ These $E$'s are intended to show sampling error, and do not address sampling bias (e.g., people with particular views refusing to participate) Also, If $p<.3$ or $>.7,$ then $E$ is smaller. Jul 29 '20 at 23:57
• Have you considered p-value or severity curves? Jul 30 '20 at 2:25
• @BruceET while the 1/sqrt(n) is a nice trick to remember, isn't it just a simplification of a theoretical interval? I believe this interval does not have true 95% coverage for smaller sample sizes.
– Alex
Jul 30 '20 at 13:16

Confidence intervals for population proportion $$p.$$ As in my Comment, if the margin of error $$E$$ in a confidence interval (CI) for a population proportion $$p$$ is $$1/\sqrt{n},$$ where $$n$$ is the sample size. So the sample size required for margin of error $$E$$ is $$n \approx 1/E^2.$$ This relationship is very simple because $$E$$ is a function of one parameter $$p.$$ Thus $$n = 2500$$ gives the confidence interval $$\hat p \pm 0.05,$$ where $$\hat p = X/n$$ is the number $$X$$ of favorable responses divided by sample size $$n.$$

E = seq(.01, .1, by .01);  n = ceiling(1/E^2)
cbind(E, n)
E     n
[1,] 0.01 10000
[2,] 0.02  2500
[3,] 0.03  1112
[4,] 0.04   625
[5,] 0.05   400
[6,] 0.06   278
[7,] 0.07   205
[8,] 0.08   157
[9,] 0.09   124
[10,] 0.10   101

For a large majority of surveys, the relationships in the table above are sufficient. For completeness, one should be aware of a couple of special cases

• For $$n$$ less than 20% of the population size and $$p = 1/2$$ these margins of error very nearly reflect inherent sampling error. (Slightly different results might result from a different sample of the same size chosen at random from the population.) For samples larger than 20% of the population size the required $$n$$ is a somewhat smaller, but surveys with $$n$$ greater than 20% of population size are rare.

• If $$p < .3$$ or $$p > .7,$$ then the sample size needed for a given $$E$$ is also somewhat smaller. For a 95% CI, $$E = 1.96\sqrt{\frac{p(1-p)}{n}}.$$ when $$p=1/2,$$ we get the largest $$E$$ for a given $$n.$$ Notice that $$(0.5)(1-0.5) = 0.25,$$ while $$(0.3)(0.7) = 0.21$$ is not much different. For $$p$$ very near to $$0$$ or $$1$$ the margin of error $$E$$ for a given $$n$$ decreases, but then issues other than sampling error may lead to difficulties conducting a survey.

Confidence intervals for population mean $$\mu.$$ For $$n$$ as large as in a typical survey, the formula for a 95% CI for unknown population $$\mu$$ is of the form $$\bar X \pm 1.96\sigma/\sqrt{n},$$ where the point estimate of $$\mu$$ is $$\bar X= \frac 1n\sum_{i=1}^n X_i.$$ Ordinarily, $$n$$ is sufficiently large that, the population standard deviation $$\sigma$$ can be estimated by the sample standard deviation $$S = \sqrt{{\frac{1}{n-1}}\sum_{i=1}^n (X_i = \bar X)^2}.$$

Thus $$E = 1.96\frac{\sigma}{\sqrt{n}}$$ or $$n \approx \left(\frac{2\sigma}{E}\right)^2.$$ Beyond the confidence level 95%, we see that two value need to be known or approximated in order to determine the sample size $$n.$$ First, we need a value for the population standard deviation $$\sigma$$ and the size of the margin of error $$E$$ relative to $$\sigma.$$

For example, if we believe $$\sigma = 3$$ and we are satisfied to have $$E = \sigma/3,$$ then the required sample size is $$n = \left(\frac{2\sigma}{E}\right)^2 = \left(\frac{2(3)}{3/3}\right)^2 = 6^2= 36.$$

The task now is to find reasonable ways to estimate $$\sigma$$ and then $$E$$ in terms of $$\sigma.$$ This may not be as difficult as it sounds at the start.

• If we want to know the mean height $$\mu$$ in the population of students at a particular university, we can look at height information online to find that $$\sigma \approx 3.5$$in is a reasonable value. Or we may guess that a span of heights that includes 95% of students is about $$14$$in, which could run from 5"0" to 6'2". Such a span divided by $$4$$ also gives $$\sigma = 3.5.$$ Having a CI of width $$2E =2$$in, might seem OK. So we would need $$n = \left(\frac{2(3.5)}{1}\right)^2 = 49$$ subjects.

• Suppose a certain pharmaceutical drug can be sold if is within 85% to 115% of nominal potency. For pills marked to contain 100mg of the drug. It is known that $$\sigma = 4$$mg is the current manufacturing capability (which should keep pills well within the 85ng to 115mg limits). We want to assay drugs sampled from pharmacies to see what average potency customers are getting. We want $$E = \sigma/5,$$ which is a company standard. Then $$n=\left(\frac{2(4)}{4/5}\right)^2 = 100$$ pharmacies.

Note: For a discussion of $$n$$ needed for a specified power of a two-sample test in terms of effect size see this link

• Thank you for the very thorough answer. My question (attempted) to be more about whether I can actually use these common intervals. Say I have 20 different groups with $n$ ranging uniformly from 5 to 100. My colleagues are looking for an exact cutoff (regardless of the statistic). How do I address this loaded question to a non-statistician? The CI for $\bar{X}$ that you give is approximately correct once the CLT kicks in. If $X$ follows a normal distribution, then $\bar{X}$ follows is assumed to follow a normal distribution with smaller $n$ then if, say, $X$ was some gamma distribution.
– Alex
Jul 30 '20 at 13:26
• Single exact 'cut off' with $n$s ranging from 5 to 100 will be difficult to justify. // 30 Heads in 100 tosses is pretty good evidence of a biased coin, 3 Heads in 10 tosses is uninformative. Jul 30 '20 at 16:11