How to test whether the difference in difference between means is significantly different? (with example) Basically I have 


*

*Four groups (2-by-2) with sample means $\overline{x_{1,A}}$, $\overline{x_{2,A}}$, $\overline{x_{1,B}}$ and $\overline{x_{2,B}}$, standard deviations $s_{1,A}$, $s_{2,A}$, $s_{1,B}$ and $s_{2,B}$ with sample sizes $n_A$ and $n_B$ (i.e. $n_{1,A}=n_{2,A}$ and $n_{1,B}=n_{2,B}$).


I calculate the differences between the two means as $d_{1-2, A} =  \overline{x_{1,A}}-\overline{x_{2,A}}$ and $d_{1-2, B} =  \overline{x_{1,B}}-\overline{x_{2,B}}$.
My objective is to test whether $d_{1-2, A}$ is statistically different from $d_{1-2, B}$, corresponding to a two sided test.
My suggestion would be: $\frac{d_{1-2, A}-d_{1-2, B}}{ [s_{1-2, A}^2/n_A] + [s_{1-2, B}^2/n_B] }$.
The problem is that I don't know how to calculate the standard deviation of this difference, i.e. $s_{1-2, A}$ and $s_{1-2, B}$.
So my questions are basically:


*

*Do you agree with my procedure?

*How do I calculate $s_{1-2, A}$ and $s_{1-2, B}$?


p.s. I tried $s_{1-2, A} = \sqrt{s_{1,A}^2 + s_{2,A}^2}$ but it didn't work.

Here is an example with numbers
$\overline{x_{1,A}} = 2.310$; $s_{1,A}=0.865$; $n_A = 16753$
$\overline{x_{1,B}} = 2.403$; $s_{1,B}=0.897$; $n_B = 5378$
$\overline{x_{2,A}} = 2.993$; $s_{2,A}=1.002$; $n_A = 16753$
$\overline{x_{2,B}} = 3.108$; $s_{2,B}=1.042$; $n_B = 5378$
The objective is to replicate a p-value for significane between $d_{1-2, A}$ and $d_{1-2, B}$ of $0.032$ (two sided would be my guess).
 A: You can get this result just by determining the distribution of the final random variable representing the difference-in-difference.  Each of your means has a standard error of $\frac{s_{i,g}}{\sqrt{n_g}}$, for $i \in \{1, 2\}$ and $g \in \{A, B\}$.  That implies that the true means (call them $X_{i,g}$) are distributed as $X_{i,g} \sim \mathcal{N}(\bar{x_{i,g}},\ \frac{s_{i,g}}{\sqrt{n_g}})$.
You want to determine the distribution of the final random variable, to see how much of it crosses 0.  That is, $X_{1,A} - X_{2,A} - (X_{1,B} - X_{2,B})$.  This is just a linear combination of Gaussian random variables, so the answer is:
$$
\mathcal{N}\!\left(\bar{x_{1,A}} - \bar{x_{2,A}} - (\bar{x_{1,B}} - \bar{x_{2,B}}),\ \ \sqrt{\frac{s_{1,A}^2}{n_A} + \frac{s_{2,A}^2}{n_A} + \frac{s_{1,B}^2}{n_B} + \frac{s_{2,B}^2}{n_B}}\right)
$$
Then you can just integrate the density of the PDF that's less than 0.  If that integral is less than your significance level, then the difference is significant.
A: Kind of a cheap trick, but because the t-test is a special case of a linear regression model adjusting for a binary predictor, the analogous regression routine for what you're calculating is a test of interaction, also called effect modification.
If you have a design matrix with an intercept, 1 column of 0/1 indicators denoting membership to one of the two groups, and another column of 0/1 indicators for membership to the comparison versus referent category in each group, then the product of these two columns gives a regressor which estimates the "difference in differences" as a parameter in the regression model. 
The standard error of this term can be obtained mathematically by calculating the covariance matrix for parameter estimates: $\sigma^2 \mathbf{X}^T\mathbf{X}$. This corresponds to the T-test for equal variances. Using the sandwich based robust standard error estimator: $(\mathbf{X}^T\mathbf{X})^{-1}(\mathbf{X}^Tr^2_i\mathbf{X})(\mathbf{X}^T\mathbf{X})^{-1}$ is analagous to the t-test in unequal populations.
