For my master's thesis I ran a small-scale experiment with a 2x2 design (42 participants, ~10 per treatment). Initially the sample was supposed to be larger, and therefore a regression analysis would have helped to estimate the difference-in-difference of the two treatment variations through an interaction term. However as the sample turned out to be very small, I am now trying to figure out what ways there might be to generate a confidence interval on this DID without running a regression. I was able to (by hand) calculate the DID, but unfortunately it does not able me to make any statistical inferences.
I tried to match treatments and was able to compare means that way, but I guess I would need to repeat this process a bunch of times to account for the fact that the matching of these separate treatments is unwarranted (between-subject design). Someone told me that permutation tests may help in randomly matching and re-matching treatments to subsequently compare means of randomly matched treatments. Is anybody familiar with using permutation testing in difference-in-difference analysis with small samples?
Hope my question is clear, if not please tell me and I'll try to re-explain.