# Statistical test of increase between groups in two time periods (categorial data)

I would like to check if percentage change between the groups and time for binominal variable is significantly different. What test to use?

In Group A (n = 1000) in t0 we had a 10% share that responded "YES", in t1 had 15% share that responded "YES"-> so +50% increase of shares

In Group B (n = 100) in t0 we had a 50% share that responded "YES", in t1 had 55% share that responded "YES" -> so +10% increase of shares

I would like to check if +50% for group A and +10% for group B is statistically different

I have checked:

You can still use a DiD-like regression to calculate the four group $$\times$$ time shares. These are random variables, with standard errors and covariances, and you will need to create a nonlinear combination of them to convert to percentage points:

$$\begin{array}{rcl} \frac{\bar{Y}(g=A,t=1) - \bar{Y}(g=A,t=0)}{\bar{Y}(g=A,t=0)} - \frac{\bar{Y}(g=B,t=1) - \bar{Y}(g=B,t=0)}{\bar{Y}(g=B,t=0)} & = & \frac{0.15 - 0.1}{0.1} - \frac{0.55 - 0.5}{0.5} \\[10pt] & = & 0.5 - 0.1 \\[10pt] & = & 0.4 \end{array}$$

You will then need to apply the delta method to calculate the variances of this difference of percent changes. Here's how to do this in Stata:

. input str1 group int(t yes noobs)

group         t       yes     noobs
1. "A" 0 0 900
2. "A" 0 1 100
3. "A" 1 0 850
4. "A" 1 1 150
5. "B" 0 0 50
6. "B" 0 1 50
7. "B" 1 0 45
8. "B" 1 1 55
9. end

. sencode group, replace

. regress yes i.group##i.t [fweight=noobs], vce(robust)

Linear regression                               Number of obs     =      2,200
F(3, 2196)        =      45.97
Prob > F          =     0.0000
R-squared         =     0.1023
Root MSE          =     .34885

------------------------------------------------------------------------------
|               Robust
yes | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
group |
B  |         .4   .0509384     7.85   0.000     .3001076    .4998924
1.t |        .05   .0147613     3.39   0.001     .0210524    .0789476
|
group#t |
B#1  |  -3.10e-16   .0721246    -0.00   1.000    -.1414396    .1414396
|
_cons |         .1   .0094955    10.53   0.000      .081379     .118621
------------------------------------------------------------------------------

. margins group#t, post //coeflegend

Adjusted predictions                                     Number of obs = 2,200
Model VCE: Robust

Expression: Linear prediction, predict()

------------------------------------------------------------------------------
|            Delta-method
|     Margin   std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
group#t |
A#0  |         .1   .0094955    10.53   0.000      .081379     .118621
A#1  |        .15   .0113019    13.27   0.000     .1278365    .1721635
B#0  |         .5   .0500455     9.99   0.000     .4018585    .5981415
B#1  |        .55   .0497947    11.05   0.000     .4523504    .6476496
------------------------------------------------------------------------------

. nlcom ///
>         (pct_change_A:(_b[1.group#1.t] - _b[1.group#0.t])/_b[1.group#0.t]) ///
>         (pct_change_B:(_b[2.group#1.t] - _b[2.group#0.t])/_b[2.group#0.t]), post //coeflegend

pct_change_A: (_b[1.group#1.t] - _b[1.group#0.t])/_b[1.group#0.t]
pct_change_B: (_b[2.group#1.t] - _b[2.group#0.t])/_b[2.group#0.t]

------------------------------------------------------------------------------
| Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
pct_change_A |         .5   .1818244     2.75   0.006     .1436307    .8563693
pct_change_B |         .1    .148459     0.67   0.501    -.1909743    .3909743
------------------------------------------------------------------------------

. nlcom ///
>         (A_vs_B_pp_change:_b[pct_change_A] - _b[pct_change_B]), post //coeflegend

A_vs_B_pp_~e: _b[pct_change_A] - _b[pct_change_B]

----------------------------------------------------------------------------------
| Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-----------------+----------------------------------------------------------------
A_vs_B_pp_change |         .4   .2347343     1.70   0.088    -.0600707    .8600707
----------------------------------------------------------------------------------

. test A_vs_B_pp_change = 0

( 1)  A_vs_B_pp_change = 0

chi2(  1) =    2.90
Prob > chi2 =    0.0884


The last command tests the null that the observed percentage point difference equals zero. The p-value is 8.8%, so we would fail to reject the null under the usual significance threshold of 5%. In other words, the observed difference could just be noise.

The above assumes that you are drawing new samples at each time rather than following the same users over time. If that is incorrect, you must cluster the standard errors by user in the regression. This will most likely raise the p-value since you have less data when you follow the same people over time. This requires a bit more than the summary statistics for me to do on your behalf, since I would need to know the number of transitions in each group:

group transition noobs
A     0->0       ?
A     0->1       ?
A     1->1       ?
A     1->0       ?
B     0->0       ?
B     0->1       ?
B     1->1       ?
B     1->0       ?