1
$\begingroup$

I am conducting an experiment in which I am testing the effect of some treatment, $X$, on some result $Y$. Every subject in both the treatment and control group is being measured multiple times (at the end of every month). I have baseline measurements of some covariate $Z$ of all subjects from the period before the experiment started. To analyse the results I am considering the following model

$$ Y = X + Z + T + XZ + XT $$

where $T$ represents a time dummy variable with $k-1$ levels (assuming every subject was measured at $k$ moments).

In R I implemented this using the following syntax:

model <- lm(Y ~ X*Z + X*T, data = data)

My main question of interest is knowing if the treatment $X$ has a significant effect on the result $Y$.

If I run

summary(model)

I might get something resembling this

   Coefficients:
                        Estimate Std. Error t value Pr(>|t|)    
   (Intercept)            3.1999377  0.0007619  4199.7   <2e-16 ***
   treatment1            -1.1999458  0.0010776 -1113.6   <2e-16 ***
   month2                 3.2000275  0.0010776  2969.7   <2e-16 ***
   month3                 6.4000800  0.0010776  5939.4   <2e-16 ***
   z                     -0.0997206  0.0004399  -226.7   <2e-16 ***
   treatment1:month2     -1.2004811  0.0015239  -787.8   <2e-16 ***
   treatment1:month3     -2.4014082  0.0015239 -1575.8   <2e-16 ***
   treatment1:z           0.0990937  0.0006222   159.3   <2e-16 *** 

    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

From this table I cannot conclude that the treatment had a significant impact on the dependent variable, because the significance of the treatment1 variable is only for (amongst other) $z = 0$ (the interactions terms are also making it difficult to estimate the effect of the treatment)

I could run the ANOVA function from the car package

Anova(model)

to summarize the influence of each factor over all of its levels. This gives me

                              Sum Sq     Df  F value    Pr(>F)    
treatment                    45613      1    11.8485  0.0005771 ***
z                            1474486    1    383.0154 < 2.2e-16 ***
month                        4621850    5    240.1163 < 2.2e-16 ***
treatment:z                  269        1    0.0699   0.7914305    
treatment:month              169658     5    8.8142   2.243e-08 ***

Is it possible to conclude from this table that the treatment had a significant effect on the result? Or are there additional steps / tests I need to perform to be able to conclude that?

Thanks in advance for the help, let me know if additional clarification is needed :)

$\endgroup$
3
$\begingroup$

What you want is to estimate the marginal effect of treatment at each time point. You can use the margins package in R to do this (though it's easier in Stata). You want to marginalize over z but hold month constant at each value you observe.

Note that your analysis violates an assumption, which is that the errors are uncorrelated with each other. Here, because you have different measurements from the same person, you need to account for the fact that the errors will be more similar for observations within a person than across people. You can look into using a multilevel model or generalized estimating equations to accomplish this. You can also run separate analyses for the outcome at each time period. Putting them all in one model makes certain assumptions that you may not intend (homogeneity of variance across time, for example).

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. The multilevel model approach can be taken by using the lme package and writing R lme(Y ~ X*Z + X*T, random = ~ 1 | id, data = data)correct? Estimating the marginal effect of the treatment at each time point makes sense. Is there a way to combine the marginal effect at each time point into an 'aggregated' marginal effect (if that makes sense). For example, suppose we measure each subject three times, we have significant effect at time point 1 and 3, with the effect being positive at point 1 and negative at point 3. What can we conclude from this? Thanks :) $\endgroup$
    – DAS
    Oct 19 '19 at 21:56
  • 1
    $\begingroup$ I'm not too familiar with MLM in R, so I can't verify if that code is correct. You can definitely compute a marginal effect averaging across time points (you could just take the average of the estimated effects), although I don't think that would be a meaningful estimate (all that really matters is the outcome at the end, no?). $\endgroup$
    – Noah
    Oct 20 '19 at 5:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.