I am trying to compute a p-value of a difference-in-difference estimator, where the null hypothesis is that the difference-in-difference is 0, and the alternative hypothesis is that the difference-in-difference is not equal to 0.
I've been thinking about this using a bootstrap approach and I don't understand why a simple approach (which I'll describe below wouldn't work). For context, a few (but non exhaustive) posts I've been reading are: Computing p-value using bootstrap with R and Non-parametric bootstrap p-values vs confidence intervals (remains unanswered but I think is the fundamental question I am also trying to ask).
So the procedure I'm thinking of is:
- Sample data B = 10000 times with replacement
- For each b in B, compute the diff-in-diff statistic, call id $d$ and save each $d$.
- To estimate the standard error you take the standard deviation resulting from the distribution of $d$'s. To compute the 95% confidence interval you take the interval (2.5 percentile, 97.5 percentile).
- If this 95% CI intersects 0, then the result is not significant at the $\alpha = 0.05$ level. If the 95% CI does not intersect 0, compute lower and lower CI until it does intersect 0. Suppose 80% CI is when it barely intersects 0, then $p = 0.2$.
Why doesn't the above logic work? I found this answer also helpful: https://stat.ethz.ch/pipermail/r-help//2009-April/387811.html which seems to suggest that the above logic would work if the assumption of translation invariance holds (which I'm not sure if I fully understand what that means).
Is there anything obviously wrong with the above?
Or maybe an even simpler approach from (3) above you have an estimate of the standard error. You can also estimate the difference-in-difference on the entire estimate for your point estimate. Then just calculate the z-score from 0 by taking the difference-in-difference estimator and dividing it by the standard error. What goes wrong there?
