0
$\begingroup$

I am comparing the drug exposures across two different groups, consisting of 1000 simulated drug exposures per group. Drug exposures are continuous variables following a normal distribution.

I want to know if different doses yield a statistically significant difference in mean drug exposure across the two groups. I am observing that even if I calibrate "artificially" the doses to generate very similar mean exposures in both groups, all the statistical tests will always return very low p-values despite the very low difference in the groups' means. I guess this is due to the very large sample size (n = 1000 per group).

However, if I reduce the sample size (to 50 virtual drug exposures, let's say) the exposure is very sensitive to the sampling procedure because the samples are taken from a distribution with high standard deviation compared to the mean, and repeating the same analysis on different datasets can give very different means in exposure.

Is this a case where I should focus more on the "biological relevance" of the difference rather than the significance of such difference? Can you suggest a different approach to judging the relevance of the difference based on robust criteria?

$\endgroup$
7
  • 4
    $\begingroup$ You can test that the difference in means is $>2$ or $<-1$, whatever a meaningful effect size is. You don't have to test that the difference is zero. $\endgroup$
    – Dave
    Jun 25 '20 at 14:29
  • $\begingroup$ D'uh, the p-value is just a reflection of the sample size :) $\endgroup$
    – AdamO
    Jun 25 '20 at 16:59
  • 1
    $\begingroup$ @Dave: This is the right answer but rather than testing for the difference in means being bigger than 2 (or whatever) it would make more sense just to focus on getting a confidence interval for the difference in means. Thats the quantity that ultimately matters, not a test with a fairly arbitrary cut-off point. $\endgroup$
    – James
    Jun 25 '20 at 18:11
  • 1
    $\begingroup$ @James The confidence level of a confidence level also is an arbitrary decision (complement of the $\alpha$-level), and confidence interval width is determined, partly, by sample size, just like a p-value is. $\endgroup$
    – Dave
    Jun 25 '20 at 18:16
  • 1
    $\begingroup$ @Dave: thats true, but knowing that a parameter is in the range [3.96,4.52] with confidence 0.95 is a lot more informative and useful than knowing that its "greater than 3 with p < 0.05", even if one statement can be deduced from the other. $\endgroup$
    – James
    Jun 25 '20 at 18:34
0
$\begingroup$

When "large" samples are necessary.

Binary data. In clinical trials for a drug, key responses are often binary---improved or not. So you're making a comparison of the number of improvements in the treatment group (with drug) and the control group (without drug). For testing such binary responses, $n=1000$ is often not too large.

Suppose you have 802 improvements among 1000 in a treatment group in a clinical trial and 765 improvements among 1000 in a control group, then the usual test of binomial proportions (as implemented in R) gives the result below. (No continuity correction is needed for sample sizes as large as 1000, so I 'turned off' corrections with parameter cor=F.)

prop.test(c(802,765), c(1000,1000), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(802, 765) out of c(1000, 1000)
X-squared = 4.0353, df = 1, p-value = 0.04456
alternative hypothesis: two.sided
95 percent confidence interval:
 0.0009360768 0.0730639232
sample estimates:
prop 1 prop 2 
0.802  0.765 

We have a marginally significant result with P-value $0.045 < 5\%.$ Also, a 95% confidence interval for the improvement with the drug is $(0.001, 0.073).$ Furthermore, as a practical matter, we are left to wonder whether an increase in improvements from 76.5% (without drug) to 80.2% (with) is good enough to justify marketing the drug.

Large sample sizes for binary data are not uncommon. A very rough rule from public opinion polling, where percentages 'in favor' are often near 50%, is that a sample of size $n$ gives a 95% confidence interval with margin of sampling error $\pm 1/\sqrt{n},$ so that it takes $n=2500$ subjects in a poll to get a margin of error of magnitude $0.02 = 2\%.$ (For $p \approx 1/2,$ the approximate formula is $1.96\sqrt{\frac{1/2(1-1/2)}{n}} \approx 1/\sqrt{n}.)$

Normal data. However, you say you are looking at an effect that is continuous and normally distributed. Roughly speaking, the number of subjects required to have 90% success, using a 2-sided, one-sample t test at significance level $\alpha = 5\%$ in detecting an effect of a certain size $\Delta,$ depends on the ratio $\Delta/\sigma,$ where $\sigma$ is the population standard deviation.

Here is a printout from Minitab's 'power and sample size' procedure. If $\sigma = 10,$ it shows sample sizes required to detect differences $\Delta = 1, 2, 3.$ A sample size of 1000 is required only if $\Delta/\sigma$ is as small as $0.1.$

Power and Sample Size 

1-Sample t Test

Testing mean = null (versus ≠ null)
Calculating power for mean = null + difference
α = 0.05  Assumed standard deviation = 10

            Sample  Target
Difference    Size   Power  Actual Power
         1    1053     0.9      0.900091
         2     265     0.9      0.900418
         3     119     0.9      0.900761

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.