1
$\begingroup$

I have two data sets. Both contain survey data where participants had to forecast the return of the same stocks. I ran two regressions one for group 1 and one for group 2 looking like this

fixed_model1 <- plm(forecast~ IV1 + IV2 + IV3, data=df1, index=c("user_id", "stock_name"), model="within")

 coeftest1 <- coeftest(fixed_model1 , vcov. = vcovHC, cluster="group") 


coeftest1
t test of coefficients:

                  Estimate Std. Error  t value  Pr(>|t|)    
IV1                 -5.8316     10.3610  -0.1347    0.1458    
IV2                254.9531   239.9216   1.3476    0.2906    
IV3               1733.7329   264.7025   6.4395 1.122e-10 ***

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


fixed_model2 <- plm(forecast~ IV1 + IV2 + IV3, data=df2, index=c("user_id", "stock_name"), model="within")

 coeftest2 <- coeftest(fixed_model2 , vcov. = vcovHC, cluster="group")

coeftest2
t test of coefficients:

                  Estimate Std. Error  t value  Pr(>|t|)    
IV1                 -1.1241     9.0413  -0.1453  0.84811
IV2                 296.3413    235.2937  1.1413  0.35113
IV3                -1756.2563   250.2423 -7.5352 1.1145e-13 ***

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Now I would like to test if the slope estimates of IV1, IV2, IV3 are significantly different between both groups. Is there a test I can use in R?

$\endgroup$
2
  • $\begingroup$ one way to do this (maybe the standard way) is to combine the two data sets (or not split them in the first place) and estimate the significance of interactions between group and IVx. $\endgroup$
    – Ben Bolker
    Jul 28, 2019 at 23:54
  • $\begingroup$ how would I do this? I already have the data sets combined and dummy variable for both studies (1,2). Would I simply define my IVs as IV_1*1 + IV_1*2 + IV_2*1 + IV_2*2 etc.? $\endgroup$
    – Jj Blevins
    Jul 28, 2019 at 23:56

2 Answers 2

2
$\begingroup$

One way to do this (maybe the standard way) is to combine the two data sets (or not split them in the first place) and estimate the significance of interactions between group and IVx.

I don't use plm much if at all, but the formula

 forecast~ group*(IV1 + IV2 + IV3)

should produce main effects of IVx that correspond to the estimates for group 1 as well as terms with names like group2:IV1 that represent the between-group differences in effects.

Note that the results will differ slightly from those you get when fitting the two groups separately, as the combined model makes the additional assumption that the residual variance is the same for both groups ...

Here are the results you quoted for your combined model (I'm including an effect of study, which you should have seen in the output ...

            Estimate Std. Error t value Pr(>|t|) 
(Intercept)        ?          ?       ?        ?
     study         ?          ?       ?        ?
       IV1   0.69991   17.08859  0.0410   0.9673 
       IV2   4.86949    9.51997  0.5115   0.6091 
 study:IV1 -14.58480   10.93147 -1.3342   0.1823 
 study:IV2   6.22122    5.88201  1.0577   0.2904
  • (Intercept): expected value for group 1 at IV1=IV2=0
  • study: expected difference between group 1 and group2 at IV1=IV2=0
  • IV1: effect of IV1 in group 1
  • IV2: effect of IV2 in group 1
  • study:IV1: difference between groups in effect of IV1
  • study:IV2: ditto, for IV2

So for statistical testing the last two p-values are what you're interested in. Note that the standard error of the IV1 parameter (effect of IV1 in group 1), and the difference between groups in IV1, and the standard error of the between-group difference in IV1, are all large. You should not say "there was no difference in the effect of IV1 between the groups" (actually, you should never say that): you should say "the estimated effect of IV1 in group 1 was small but highly uncertain: the estimated effect of IV2 in group 2 appears much lower, but the difference is also highly uncertain, so we can't make any clear statements about the direction of the difference" (or something like that ...)

$\endgroup$
8
  • $\begingroup$ Do I need to define the group_variables as factors for this to work? $\endgroup$
    – Jj Blevins
    Jul 29, 2019 at 2:12
  • $\begingroup$ I tried it, unfortunately it didn't work. $\endgroup$
    – Jj Blevins
    Jul 29, 2019 at 12:36
  • 1
    $\begingroup$ what does "didn't work" mean ... ? $\endgroup$
    – Ben Bolker
    Jul 29, 2019 at 14:29
  • $\begingroup$ Sorry, for not being specific. I meant that I do not understand the output: Estimate Std. Error t value Pr(>|t|) IV1 0.69991 17.08859 0.0410 0.9673 IV2 4.86949 9.51997 0.5115 0.6091 study:IV1 -14.58480 10.93147 -1.3342 0.1823 study:IV2 6.22122 5.88201 1.0577 0.2904 $\endgroup$
    – Jj Blevins
    Jul 29, 2019 at 14:32
  • $\begingroup$ Could you telle me how to interpret these results? $\endgroup$
    – Jj Blevins
    Jul 30, 2019 at 0:04
0
$\begingroup$

Bootstrap your coefficients. Randomly sample from each of your populations, fit your regression model repeatedly on the random sample, giving you a distribution of your coefficient for each population.

You can then validate your hypothesis (statistically distinct distributions for each coefficient within each population) using a confidence interval approach based on whatever p-value you want (.01, .05, etc.)

Not an R person, so you'll have to code it yourself.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.