# Statistical significance between two percentage change

I have two groups (test and control) and I wanted to determine the impact of a certain initiative on the test group. The comparison would be 4-month before the initiative and 4-month after. The starting baselines for the groups are slightly different so to calculate the impact use percentages. If the test group improves by 5% after the initiative while the control group changes by 1%. How do I determine whether the 5% change is statistically significance compared to the 1% change in the control group?

• Are you still designing this study? If so, I recommend more than two time points. – Peter Flom Dec 13 '12 at 20:43
• Assuming your sample size is decent, the standardized difference between proportions within a single group is approximately normally distributed; that's the basis of the basic two-sample test of proportions. So, the difference between those (scaled) mean differences in also approximately normally distributed, from which you could base a test. A better alternative (that respects the pairing of the data) might be to compare the proportion who moved from the "bad" to 'good" category, or work out a strategy based on logistic regression (this data structure is essentially a 2x2x2 contigency table). – gammer Mar 26 '17 at 16:46

If you have measured improvement for every member in every group, you just need to perform a t-test to test if the mean improvement percentage is different in the test group and the control group.

I ran a little simulation using R to see if you'd get different results using percentage of change vs scaled scores. The results are interesting...

### Generate some data

I used N=40 for both groups, variance approximately equal

set.seed(32453513)
test_group <- data.frame(before = rnorm(n = 40, mean = 5.5, sd = 1.8),
after  = rnorm(n = 40, mean = 5.775, sd = 1.7))
ctrl_group <- data.frame(before = rnorm(n = 40, mean = 5.8, sd = 1.8),
after  = rnorm(n = 40, mean = 5.858, sd = 1.7))


### Test differences on raw scores

test_group$raw_diffs <- with(test_group, after - before) ctrl_group$raw_diffs <- with(ctrl_group, after - before)
t.test(x = test_group$raw_diffs, y = ctrl_group$raw_diffs)

## Welch Two Sample t-test
##
## data:  test_group$raw_diffs and ctrl_group$raw_diffs
## t = -0.23249, df = 77.902, p-value = 0.8168
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -1.1182797  0.8844153
## sample estimates:
## mean of x mean of y
## 0.6738046 0.7907368


### Test differences on scaled (NOT centered) scores

test_group_std <- with(test_group, data.frame(before = scale(before, center = FALSE),
after  = scale(after,  center = FALSE)))
ctrl_group_std <- with(ctrl_group, data.frame(before = scale(before, center = FALSE),
after  = scale(after,  center = FALSE)))

test_group_std$diff <- with(test_group_std, after - before) ctrl_group_std$diff <- with(ctrl_group_std, after - before)

t.test(x = test_group_std$diff, y = ctrl_group_std$diff)

## Welch Two Sample t-test
##
## data:  test_group_std$diff and ctrl_group_std$diff
## t = -0.14204, df = 77.807, p-value = 0.8874
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.1755982  0.1522108
## sample estimates:
##   mean of x   mean of y
## 0.005331736 0.017025437


### Test differences on pct. change

test_group$pct_diffs <- with(test_group, (after - before)/before) ctrl_group$pct_diffs <- with(ctrl_group, (after - before)/before)
t.test(x = test_group$raw_diffs, y = ctrl_group$raw_diffs)

## Welch Two Sample t-test
##
## data:  test_group$raw_diffs and ctrl_group$raw_diffs
## t = -0.23249, df = 77.902, p-value = 0.8168
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -1.1182797  0.8844153
## sample estimates:
## mean of x mean of y
## 0.6738046 0.7907368


This makes it clear that working with raw scores is no different from working with pct. change. The scaling thus would be the method to be used to eliminate the bias coming from the difference on baseline scores.

I'm not entirely convinced this is the absolute best way to go, maybe statisticians call clue us in here...