Statistical significance between two percentage change I have two groups (test and control) and I wanted to determine the impact of a certain initiative on the test group.  The comparison would be 4-month before the initiative and 4-month after.  The starting baselines for the groups are slightly different so to calculate the impact use percentages.  If the test group improves by 5% after the initiative while the control group changes by 1%.  How do I determine whether the 5% change is statistically significance compared to the 1% change in the control group?
 A: If you have measured improvement for every member in every group, you just need to perform a t-test to test if the mean improvement percentage is different in the test group and the control group.
A: I ran a little simulation using R to see if you'd get different results using percentage of change vs scaled scores. The results are interesting...
Generate some data
I used N=40 for both groups, variance approximately equal
set.seed(32453513)
test_group <- data.frame(before = rnorm(n = 40, mean = 5.5, sd = 1.8),
                         after  = rnorm(n = 40, mean = 5.775, sd = 1.7))
ctrl_group <- data.frame(before = rnorm(n = 40, mean = 5.8, sd = 1.8),
                         after  = rnorm(n = 40, mean = 5.858, sd = 1.7))

Test differences on raw scores
test_group$raw_diffs <- with(test_group, after - before)
ctrl_group$raw_diffs <- with(ctrl_group, after - before)
t.test(x = test_group$raw_diffs, y = ctrl_group$raw_diffs)

## Welch Two Sample t-test
##
## data:  test_group$raw_diffs and ctrl_group$raw_diffs
## t = -0.23249, df = 77.902, p-value = 0.8168
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -1.1182797  0.8844153
## sample estimates:
## mean of x mean of y 
## 0.6738046 0.7907368

Test differences on scaled (NOT centered) scores
test_group_std <- with(test_group, data.frame(before = scale(before, center = FALSE), 
                                              after  = scale(after,  center = FALSE)))
ctrl_group_std <- with(ctrl_group, data.frame(before = scale(before, center = FALSE), 
                                              after  = scale(after,  center = FALSE)))

test_group_std$diff <- with(test_group_std, after - before)
ctrl_group_std$diff <- with(ctrl_group_std, after - before)

t.test(x = test_group_std$diff, y = ctrl_group_std$diff)

## Welch Two Sample t-test
##
## data:  test_group_std$diff and ctrl_group_std$diff
## t = -0.14204, df = 77.807, p-value = 0.8874
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.1755982  0.1522108
## sample estimates:
##   mean of x   mean of y 
## 0.005331736 0.017025437 

Test differences on pct. change
test_group$pct_diffs <- with(test_group, (after - before)/before)
ctrl_group$pct_diffs <- with(ctrl_group, (after - before)/before)
t.test(x = test_group$raw_diffs, y = ctrl_group$raw_diffs)

## Welch Two Sample t-test
## 
## data:  test_group$raw_diffs and ctrl_group$raw_diffs
## t = -0.23249, df = 77.902, p-value = 0.8168
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -1.1182797  0.8844153
## sample estimates:
## mean of x mean of y 
## 0.6738046 0.7907368 

This makes it clear that working with raw scores is no different from working with pct. change. The scaling thus would be the method to be used to eliminate the bias coming from the difference on baseline scores.
I'm not entirely convinced this is the absolute best way to go, maybe statisticians call clue us in here...
