For my master's thesis I ran a small-scale experiment with a 2x2 design (42 participants, ~10 per treatment). Initially the sample was supposed to be larger, and therefore a regression analysis would have helped to estimate the difference-in-difference of the two treatment variations through an interaction term. However as the sample turned out to be very small, I am now trying to figure out what ways there might be to generate a confidence interval on this DID without running a regression. I was able to (by hand) calculate the DID, but unfortunately it does not able me to make any statistical inferences.

I tried to match treatments and was able to compare means that way, but I guess I would need to repeat this process a bunch of times to account for the fact that the matching of these separate treatments is unwarranted (between-subject design). Someone told me that permutation tests may help in randomly matching and re-matching treatments to subsequently compare means of randomly matched treatments. Is anybody familiar with using permutation testing in difference-in-difference analysis with small samples?

Hope my question is clear, if not please tell me and I'll try to re-explain.

  • $\begingroup$ If you assigned treatment randomly, then DiD has great properties - regardless of the sample size. Besides that, it makes no difference whether you use regression or not - the difference in difference estimates is the same, and so is the standard error. So even if you choose not to report the results in a regression table, you can still use it to calculate a confidence band. $\endgroup$ – Repmat Jul 12 '15 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.