2
$\begingroup$

We want to calculate sample size for detecting the difference between mean depression scores of two groups of patients after three months of treatment - i.e. those who received a specialized mental health intervention ("treatment group") and those received routine treatment ("control group"). These patients are randomly assigned to the two groups. Also, the mean scores are differences between the baseline (pre-treatment) and end-line (post-treatment or at 3 months) scores of the respective groups. So, we wanted to calculate the sample size for detecting the difference in differences. Which is the best approach to follow for this situation?

$\endgroup$

2 Answers 2

2
$\begingroup$

Assuming treatment and control 3-month differences are nearly normally distributed, you want a power and sample size computation for a two-sample t test.

In order to get the sample size for each group you need the following:

  • Approximate variance of data in the two groups,
  • Significance level of the test [5% ?],
  • Size of difference you want to detect, power (probability of detecting a difference of that size, if it exists) [95% ?],
  • Whether the test is one or two-sided.

Many statistical software programs (and some Internet sites) have procedures for doing the computations. For example, here is output from a power and sample size procedure in Minitab statistical software.

Power and Sample Size 

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 10


            Sample  Target
Difference    Size   Power  Actual Power
         5      64    0.80      0.801460
         5      86    0.90      0.903230
         5     105    0.95      0.950129

The sample size is for each group.

enter image description here

Notes: (1) If the ratio of the difference to detect to the standard deviation is the same as above $(5/10 = 0.5),$ the required sample size will be the same (for a two-sided test with the same significance level and power).

(2) For cases where the required sample size is large, a rough approximate formula is $n \approx \frac{2\sigma^2(z_{\beta} + z_{\alpha/2})^2}{\Delta^2},$ where $\Delta$ is the difference between population means to detect.

For the parameters above, this approximate formula gives $n\approx 104,$ which is not far from the Minitab value $(n=105)$ for power $.95 = 1 - \beta.$

2*10^2*(1.645+1.96)^2/5^2
[1] 103.9682

(3) A balanced design with equal sample sizes $n_1=n_2,$ is most efficient, so most 'power and sample size' procedure assume equal sample sizes.

(4) In practice, it is best to use a Welch 2-sample t test (which does not require that the two populations have the same variance $\sigma^2)$, unless you have prior experience with similar measurements and are reasonably sure population variances are equal. The procedure from Minitab (and many other programs) is for a pooled 2-sample t test, which does require equal population variances.

(5) Power for the Welch test can be simulated as shown below (in R) for a test with unequal variances (and also perhaps unequal sample sizes).

Below, sample sizes are $n_1=60, n_2=70$ (bracketing Minitab's $n_1=n_2=64)$ and population standard deviations are $\sigma_1 = 8, \sigma_2=12,$ (bracketing my input to Minitab with $\sigma_1 = \sigma_2 = 10.$ [In R, t.test is the Welch test unless the additional parameter var.eq=T is used.]

According to the simulation, the power of the Welch test in these circumstances is about $0.8 = 80\%,$ similar to Minitab's power $80\%$ above.

set.seed(121)
pv = replicate(10^5, 
               t.test(rnorm(60, 100, 8), rnorm(70, 105, 12))$p,val)
mean(pv <= .05)
[1] 0.8003      # aprx power of Welch test at 5% level 

summary(pv)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
0.0000000 0.0006006 0.0054053 0.0457570 0.0333773 0.9997453 
$\endgroup$
3
  • $\begingroup$ Thanks Bruce for answering the question so promptly and comprehensively! However, I guess our question was slightly different - We want to calculate the number of participants required to measure the difference in differences between the baseline and endline mean scores of participants of the two groups. We will explain the "difference in differences" below - - the before-after difference in mean depression scores for the treatment group (B-A). - the before-after difference in mean depression scores for the comparison group (D-C) - the difference between the above differences in depression sco $\endgroup$ Jan 6, 2021 at 11:17
  • $\begingroup$ I believe I did understand and answer your question. Your two samples will be the 3-month differences for treatment group and 3-month differences for control group, and the variance is the variance of the differences. // If you have an ANOVA in mind also testing whether there is any change of treatment or control subjects over 3 months, that is fine. In the end looking a differences in differences comes down to a 2-sample test that's equivalent to a t test. If sample sizes are big enough for that final comparison, they will also be big enough for what amount to paired t tests for 3-mo diff. $\endgroup$
    – BruceET
    Jan 6, 2021 at 11:39
  • $\begingroup$ Thanks Bruce, so shall we use either of these methods in GPower (or any equivalent software) to calculate sample size i.e. 1) ANOVA: Repeated measures, between factors 2) Means: Difference between two independent means (Two groups) $\endgroup$ Jan 6, 2021 at 12:15
2
$\begingroup$

A parallel-group randomized trial is design to compare parallel groups, not to compare change from baseline. You need to be using ANCOVA with Y=post score and X=baseline score. Change from baseline assumes linearity and a slope of 1.0 for post-pre. In depression studies I've seen strong nonlinearity. The most general analysis that is powerful is a generalization of the Wilcoxon test: the proportional odds ordinal logistic model with Y=post score and using a cubic spline function to adjust for X. More details are here.

That being said, for planning purposes, when we don't have pilot data that allows us to do ANCOVA, we assume linearity and a slope of 1.0 and use the unpaired t-test on change from baseline for power calculation when there are no intermediate measurements between baseline and 3m. This will underestimate power because the mean squared error assuming a slope of 1.0 will be higher than the mean squared error that estimates the slope from the data. But it is better to use the two-sample t-test to analyze change from baseline than it is to get a much bigger mean squared error by using a two-sample t-test without baseline subtraction, which assumes a slope of 0.0.

$\endgroup$
1
  • $\begingroup$ Thanks Frank, we also found the following paper that may help. Would you recommend following either of the methods explained in 3.1 and 3.2, if not the ANCOVA, due to strong assumptions in the latter. ncbi.nlm.nih.gov/pmc/articles/PMC6297128/#appsec1 $\endgroup$ Jan 7, 2021 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.