Is there a way to test for the significance of the difference-in-differences in adj. R²s in Stata?

Let's say I have four subgroups: pre-treatment, pre-control, post-treatment, post-control and I want to figure out whether the difference-in-differences of the adj. R² of the same regression (e.g., R² of Y = b0 + b1*X + e) conducted for each of the four subgroups is signficant.

In other words, is (adj. R²(post-treatment) - adj. R²(pre-treatment)) - ((adj. R²(post-control) - adj. R²(pre-control)) signficant?

I read somewhere that it might be possible to bootstrap the adj. R² of each regression and then test for the difference-in-differences, but I am not sure if this would be an appropriate approach?

Thanks a lot!


Whilst it is true that you can bootstrap $\text{R}^2$ and $\overline{\text{R}}^2$ (see Ohtani (2001) "Bootstrapping $\text{R}^2$ and adjusted $\text{R}^2$ in regression analysis") the procedure you propose will result in having 4 numbers, i.e. the four $\text{R}^2$ or $\overline{\text{R}}^2$ from the regressions of $Y$ on $X$ for the treatment and control groups in the two time periods. In this case you end up like the OP of this earlier question who tried to achieve exactly that but finally figured out that it could not be done. So even if you can construct a standard error or confidence interval for each coefficient of determination itself, with four numbers there is no degree of freedom left in order to calculate a standard error for the difference in differences.

  • $\begingroup$ So if I understand correctly, I would need the standard error/confidence interval (distribution) for the differences between two groups (e.g., R² treatment - R² control) which is probably not computeable for R² (e.g., by random sampling of the differences), or am I wrong here? $\endgroup$ – user51972 Jul 12 '14 at 19:41
  • $\begingroup$ Actually you would need the standard error of the difference in differences. If you had a sequence of each R² measure this is feasible but not with only 4 numbers only. $\endgroup$ – Andy Jul 12 '14 at 20:00
  • $\begingroup$ Is it feasable to create a sequence of R² difference-in-differences by drawing random observations from the full sample? Let's say, I randomly draw a number of treatment obs. and (matched) non-treatment obs., calculate the difference-in-differences of their R², and repeat that x times to get the distribution of the difference-in-differences? $\endgroup$ – user51972 Jul 12 '14 at 23:58
  • $\begingroup$ I can see where you are coming from though I'm somewhat hesitant towards this procedure, yet I haven't found a good reason against it so far. How do you intend to obtain a standard error for the difference in differences estimate in this way? $\endgroup$ – Andy Jul 14 '14 at 0:02
  • $\begingroup$ Let's say I randomly draw a number of matched pairs in the pre-period and then calculate an R² for these observations, but seperately for each "bucket" (pre-treat; pre-control; post-trest; post-control). This way I'm able to calculate a diff-in-diff R². Repeating this procedure x times I'd get the distribution, or? $\endgroup$ – user51972 Jul 17 '14 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.