Significance of treatment effect when interactions are present

Question

I am conducting an experiment in which I am testing the effect of some treatment, $X$, on some result $Y$. Every subject in both the treatment and control group is being measured multiple times (at the end of every month). I have baseline measurements of some covariate $Z$ of all subjects from the period before the experiment started. To analyse the results I am considering the following model

$$ Y = X + Z + T + XZ + XT $$

where $T$ represents a time dummy variable with $k-1$ levels (assuming every subject was measured at $k$ moments).

In R I implemented this using the following syntax:

model <- lm(Y ~ X*Z + X*T, data = data)

My main question of interest is knowing if the treatment $X$ has a significant effect on the result $Y$.

If I run

summary(model)

I might get something resembling this

   Coefficients:
                        Estimate Std. Error t value Pr(>|t|)    
   (Intercept)            3.1999377  0.0007619  4199.7   <2e-16 ***
   treatment1            -1.1999458  0.0010776 -1113.6   <2e-16 ***
   month2                 3.2000275  0.0010776  2969.7   <2e-16 ***
   month3                 6.4000800  0.0010776  5939.4   <2e-16 ***
   z                     -0.0997206  0.0004399  -226.7   <2e-16 ***
   treatment1:month2     -1.2004811  0.0015239  -787.8   <2e-16 ***
   treatment1:month3     -2.4014082  0.0015239 -1575.8   <2e-16 ***
   treatment1:z           0.0990937  0.0006222   159.3   <2e-16 *** 

    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

From this table I cannot conclude that the treatment had a significant impact on the dependent variable, because the significance of the treatment1 variable is only for (amongst other) $z = 0$ (the interactions terms are also making it difficult to estimate the effect of the treatment)

I could run the ANOVA function from the car package

Anova(model)

to summarize the influence of each factor over all of its levels. This gives me

                              Sum Sq     Df  F value    Pr(>F)    
treatment                    45613      1    11.8485  0.0005771 ***
z                            1474486    1    383.0154 < 2.2e-16 ***
month                        4621850    5    240.1163 < 2.2e-16 ***
treatment:z                  269        1    0.0699   0.7914305    
treatment:month              169658     5    8.8142   2.243e-08 ***

Is it possible to conclude from this table that the treatment had a significant effect on the result? Or are there additional steps / tests I need to perform to be able to conclude that?

Thanks in advance for the help, let me know if additional clarification is needed :)

Answer 1

What you want is to estimate the marginal effect of treatment at each time point. You can use the margins package in R to do this (though it's easier in Stata). You want to marginalize over z but hold month constant at each value you observe.

Note that your analysis violates an assumption, which is that the errors are uncorrelated with each other. Here, because you have different measurements from the same person, you need to account for the fact that the errors will be more similar for observations within a person than across people. You can look into using a multilevel model or generalized estimating equations to accomplish this. You can also run separate analyses for the outcome at each time period. Putting them all in one model makes certain assumptions that you may not intend (homogeneity of variance across time, for example).