Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Is there a way to test for the significance of the difference-in-differences in adj. R²s in Stata?

Let's say I have four subgroups: pre-treatment, pre-control, post-treatment, post-control and I want to figure out whether the difference-in-differences of the adj. R² of the same regression (e.g., R² of Y = b0 + b1*X + e) conducted for each of the four subgroups is signficant.

In other words, is (adj. R²(post-treatment) - adj. R²(pre-treatment)) - ((adj. R²(post-control) - adj. R²(pre-control)) signficant?

I read somewhere that it might be possible to bootstrap the adj. R² of each regression and then test for the difference-in-differences, but I am not sure if this would be an appropriate approach?

Thanks a lot!

share|improve this question

1 Answer 1

up vote 2 down vote accepted

Whilst it is true that you can bootstrap $\text{R}^2$ and $\overline{\text{R}}^2$ (see Ohtani (2001) "Bootstrapping $\text{R}^2$ and adjusted $\text{R}^2$ in regression analysis") the procedure you propose will result in having 4 numbers, i.e. the four $\text{R}^2$ or $\overline{\text{R}}^2$ from the regressions of $Y$ on $X$ for the treatment and control groups in the two time periods. In this case you end up like the OP of this earlier question who tried to achieve exactly that but finally figured out that it could not be done. So even if you can construct a standard error or confidence interval for each coefficient of determination itself, with four numbers there is no degree of freedom left in order to calculate a standard error for the difference in differences.

share|improve this answer
    
So if I understand correctly, I would need the standard error/confidence interval (distribution) for the differences between two groups (e.g., R² treatment - R² control) which is probably not computeable for R² (e.g., by random sampling of the differences), or am I wrong here? –  user51972 Jul 12 at 19:41
    
Actually you would need the standard error of the difference in differences. If you had a sequence of each R² measure this is feasible but not with only 4 numbers only. –  Andy Jul 12 at 20:00
    
Is it feasable to create a sequence of R² difference-in-differences by drawing random observations from the full sample? Let's say, I randomly draw a number of treatment obs. and (matched) non-treatment obs., calculate the difference-in-differences of their R², and repeat that x times to get the distribution of the difference-in-differences? –  user51972 Jul 12 at 23:58
    
I can see where you are coming from though I'm somewhat hesitant towards this procedure, yet I haven't found a good reason against it so far. How do you intend to obtain a standard error for the difference in differences estimate in this way? –  Andy Jul 14 at 0:02
    
Let's say I randomly draw a number of matched pairs in the pre-period and then calculate an R² for these observations, but seperately for each "bucket" (pre-treat; pre-control; post-trest; post-control). This way I'm able to calculate a diff-in-diff R². Repeating this procedure x times I'd get the distribution, or? –  user51972 Jul 17 at 23:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.