Statistical comparison for large sample sizes (n>1000)

I am comparing the drug exposures across two different groups, consisting of 1000 simulated drug exposures per group. Drug exposures are continuous variables following a normal distribution.

I want to know if different doses yield a statistically significant difference in mean drug exposure across the two groups. I am observing that even if I calibrate "artificially" the doses to generate very similar mean exposures in both groups, all the statistical tests will always return very low p-values despite the very low difference in the groups' means. I guess this is due to the very large sample size (n = 1000 per group).

However, if I reduce the sample size (to 50 virtual drug exposures, let's say) the exposure is very sensitive to the sampling procedure because the samples are taken from a distribution with high standard deviation compared to the mean, and repeating the same analysis on different datasets can give very different means in exposure.

Is this a case where I should focus more on the "biological relevance" of the difference rather than the significance of such difference? Can you suggest a different approach to judging the relevance of the difference based on robust criteria?

• You can test that the difference in means is $>2$ or $<-1$, whatever a meaningful effect size is. You don't have to test that the difference is zero.
– Dave
Jun 25 '20 at 14:29
• D'uh, the p-value is just a reflection of the sample size :) Jun 25 '20 at 16:59
• @Dave: This is the right answer but rather than testing for the difference in means being bigger than 2 (or whatever) it would make more sense just to focus on getting a confidence interval for the difference in means. Thats the quantity that ultimately matters, not a test with a fairly arbitrary cut-off point. Jun 25 '20 at 18:11
• @James The confidence level of a confidence level also is an arbitrary decision (complement of the $\alpha$-level), and confidence interval width is determined, partly, by sample size, just like a p-value is.
– Dave
Jun 25 '20 at 18:16
• @Dave: thats true, but knowing that a parameter is in the range [3.96,4.52] with confidence 0.95 is a lot more informative and useful than knowing that its "greater than 3 with p < 0.05", even if one statement can be deduced from the other. Jun 25 '20 at 18:34

1 Answer

When "large" samples are necessary.

Binary data. In clinical trials for a drug, key responses are often binary---improved or not. So you're making a comparison of the number of improvements in the treatment group (with drug) and the control group (without drug). For testing such binary responses, $$n=1000$$ is often not too large.

Suppose you have 802 improvements among 1000 in a treatment group in a clinical trial and 765 improvements among 1000 in a control group, then the usual test of binomial proportions (as implemented in R) gives the result below. (No continuity correction is needed for sample sizes as large as 1000, so I 'turned off' corrections with parameter cor=F.)

prop.test(c(802,765), c(1000,1000), cor=F)

2-sample test for equality of proportions
without continuity correction

data:  c(802, 765) out of c(1000, 1000)
X-squared = 4.0353, df = 1, p-value = 0.04456
alternative hypothesis: two.sided
95 percent confidence interval:
0.0009360768 0.0730639232
sample estimates:
prop 1 prop 2
0.802  0.765


We have a marginally significant result with P-value $$0.045 < 5\%.$$ Also, a 95% confidence interval for the improvement with the drug is $$(0.001, 0.073).$$ Furthermore, as a practical matter, we are left to wonder whether an increase in improvements from 76.5% (without drug) to 80.2% (with) is good enough to justify marketing the drug.

Large sample sizes for binary data are not uncommon. A very rough rule from public opinion polling, where percentages 'in favor' are often near 50%, is that a sample of size $$n$$ gives a 95% confidence interval with margin of sampling error $$\pm 1/\sqrt{n},$$ so that it takes $$n=2500$$ subjects in a poll to get a margin of error of magnitude $$0.02 = 2\%.$$ (For $$p \approx 1/2,$$ the approximate formula is $$1.96\sqrt{\frac{1/2(1-1/2)}{n}} \approx 1/\sqrt{n}.)$$

Normal data. However, you say you are looking at an effect that is continuous and normally distributed. Roughly speaking, the number of subjects required to have 90% success, using a 2-sided, one-sample t test at significance level $$\alpha = 5\%$$ in detecting an effect of a certain size $$\Delta,$$ depends on the ratio $$\Delta/\sigma,$$ where $$\sigma$$ is the population standard deviation.

Here is a printout from Minitab's 'power and sample size' procedure. If $$\sigma = 10,$$ it shows sample sizes required to detect differences $$\Delta = 1, 2, 3.$$ A sample size of 1000 is required only if $$\Delta/\sigma$$ is as small as $$0.1.$$

Power and Sample Size

1-Sample t Test

Testing mean = null (versus ≠ null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 10

Sample  Target
Difference    Size   Power  Actual Power
1    1053     0.9      0.900091
2     265     0.9      0.900418
3     119     0.9      0.900761