# Significance of treatment effect when interactions are present

I am conducting an experiment in which I am testing the effect of some treatment, $$X$$, on some result $$Y$$. Every subject in both the treatment and control group is being measured multiple times (at the end of every month). I have baseline measurements of some covariate $$Z$$ of all subjects from the period before the experiment started. To analyse the results I am considering the following model

$$Y = X + Z + T + XZ + XT$$

where $$T$$ represents a time dummy variable with $$k-1$$ levels (assuming every subject was measured at $$k$$ moments).

In R I implemented this using the following syntax:

model <- lm(Y ~ X*Z + X*T, data = data)


My main question of interest is knowing if the treatment $$X$$ has a significant effect on the result $$Y$$.

If I run

summary(model)


I might get something resembling this

   Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)            3.1999377  0.0007619  4199.7   <2e-16 ***
treatment1            -1.1999458  0.0010776 -1113.6   <2e-16 ***
month2                 3.2000275  0.0010776  2969.7   <2e-16 ***
month3                 6.4000800  0.0010776  5939.4   <2e-16 ***
z                     -0.0997206  0.0004399  -226.7   <2e-16 ***
treatment1:month2     -1.2004811  0.0015239  -787.8   <2e-16 ***
treatment1:month3     -2.4014082  0.0015239 -1575.8   <2e-16 ***
treatment1:z           0.0990937  0.0006222   159.3   <2e-16 ***

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


From this table I cannot conclude that the treatment had a significant impact on the dependent variable, because the significance of the treatment1 variable is only for (amongst other) $$z = 0$$ (the interactions terms are also making it difficult to estimate the effect of the treatment)

I could run the ANOVA function from the car package

Anova(model)


to summarize the influence of each factor over all of its levels. This gives me

                              Sum Sq     Df  F value    Pr(>F)
treatment                    45613      1    11.8485  0.0005771 ***
z                            1474486    1    383.0154 < 2.2e-16 ***
month                        4621850    5    240.1163 < 2.2e-16 ***
treatment:z                  269        1    0.0699   0.7914305
treatment:month              169658     5    8.8142   2.243e-08 ***


Is it possible to conclude from this table that the treatment had a significant effect on the result? Or are there additional steps / tests I need to perform to be able to conclude that?

Thanks in advance for the help, let me know if additional clarification is needed :)

What you want is to estimate the marginal effect of treatment at each time point. You can use the margins package in R to do this (though it's easier in Stata). You want to marginalize over z but hold month constant at each value you observe.
• Thank you for your answer. The multilevel model approach can be taken by using the lme package and writing R lme(Y ~ X*Z + X*T, random = ~ 1 | id, data = data)correct? Estimating the marginal effect of the treatment at each time point makes sense. Is there a way to combine the marginal effect at each time point into an 'aggregated' marginal effect (if that makes sense). For example, suppose we measure each subject three times, we have significant effect at time point 1 and 3, with the effect being positive at point 1 and negative at point 3. What can we conclude from this? Thanks :)