1
$\begingroup$

I'm analyzing a randomized controlled trial. At baseline time, the outcome variable of the control group (C) was slightly higher than that of the intervention group (I), but such difference wasn't significant.

At the end of the trial (3-month time), outcomes both groups had higher values than the baseline. Within-group change among (C) was not significant but within-group change among (I) was significant. Between-group change wasn't significant.

One member of the team thought that shouldn't be the case, i.e., if (I) showed significant within-group change, that would lead to significant between-group difference.

I think I need to convince with some formal proof. That is to show that, on theory, that can happen. Can you tell some relatively simple mathematical reference for this sort of test? "Light" math is preferable though.

EDIT to take into account Frans Rodenburg's comment.

$\endgroup$
3
  • $\begingroup$ You haven't explicitly included in your question whether the difference between (C) and (I) is insignificant after 3 months, but I'll assume it was. This can happen for example if the control group has higher variance than the intervention group, rendering the comparison between them insignificant, while the before/after difference is significant for the intervention group. Is the difference still significant after correcting for 4 hypothesis tests? $\endgroup$ Commented Feb 19, 2018 at 3:28
  • $\begingroup$ Thanks. Yes the between-group change was insignificant. I think I don't understand your last sentence about the "4 hypothesis tests" bit. $\endgroup$
    – NonSleeper
    Commented Feb 19, 2018 at 4:36
  • 1
    $\begingroup$ If you perform 4 tests of significance, the real type I error rate is much higher than the chosen threshold for significance (e.g. 0.05). You have to correct for this using e.g. Bonferroni adjustment (multiply all $p$-values by 4 in this case) or FDR. See for example the Wikipedia page on multiple testing: en.wikipedia.org/wiki/Multiple_comparisons_problem $\endgroup$ Commented Feb 19, 2018 at 4:40

2 Answers 2

1
$\begingroup$

Let's assume we're using t-tests. The within-group change is a paired t-test, while the between-group change is a two sample t-test. Only one treatment showing a significant within-group change does not necessarily imply a significant between-group change. We could go through the formulas, but let's just consider a basic example.

Suppose the outcome, $Y$, at baseline is equal 0 for both groups, and the outcome at 3-months be $Y \sim N(1,\sigma^2)$ where $\sigma^2=100$ for group C and $\sigma^2=0.1$ for group I. Thus, the standard deviation of the within-group change for group I is smaller than group C, so we likely would only find significance for group I depending on our sample size. The mean difference between the two groups is the same, so we shouldn't find a significant between-group change. You can see this in the following simulation with N=10 subjects for each group.

dat <- data.frame(trt=rep(c("I","C"),each=20),
                  period=rep(rep(c("Baseline","3-month"),each=10),2),
                  outcome=c(rep(0,10),rnorm(10,mean=1,sd=0.1),
                            rep(0,10),rnorm(10,mean=1,sd=100)))

t.test(outcome~period, data=dat[dat$trt=="I",], paired=TRUE)
t.test(outcome~period, data=dat[dat$trt=="C",], paired=TRUE)
t.test(outcome~trt, data=dat[dat$period=="3-month",])
$\endgroup$
0
$\begingroup$

A possible explanation could be as Rodenburg commented (that one group has higher variance).

Another explanation could be that in Baseline one group (e.g. C) has higher mean value and after 3 months the other group has higher mean value. So, in the vertical axis: between groups it's the same difference but within groups for I there is bigger difference.

Graph

You can test your data to see if that's the case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.