# Treatment evaluation: Measuring statistically / clinically significant change in level over time

I am conducting a treatment evaluation. I am using an interrupted time series design (generalised linear mixed model; 42 monthly measurements per patient [18 pre-treatment, 24 post-treatment). I have assessed the impact of the treatment program on the level and trend of an adverse outcome for the whole sample (treatment vs no treatment) and would now like to assess whether the impact of treatment on the outcome is moderated by a statistically / clinically significant change in the provision of services. That is, is the effect of treatment "conditioned" on an increase in contact with health services.

Initially, I considered calculating change in contact using the reliable change index (e.g., baseline, mean amount of contact in X months before treatment; post-treatment, mean amount of contact in X months following treatment); however, I would really like to calculate change from the baseline level over time to the post-treatment level over time.

I have also considered comparing the baseline and post-treatment levels using another interrupted time series model, but my understanding is that the estimate for the post-treatment level will be for the time immediately following implementation of the treatment program and I am unclear how to extrapolate this to a level over time.

Question

What is an appropriate way to calculate statistically / clinically significant change in the level of a variable using multiple observations?

This does not directly get at your question, but since you seem to be ultimately interested in controlling for this change in your main ITS model, have you considered applying a time-varying covariate approach by interacting monthly amount of contact with your "time since the start of the study" variable, perhaps only in the post-intervention period (if you only found an effect on the slope in the main model)? Then you will be able to test the significance of those variables and would not have to worry about how you've specified the change.

So, if the original ITS model (without any covariates or random effects) took the form of:

$$Y_{t} = \beta _{0} + \beta _{1}T_{t} + \beta _{2}X_{t} + \beta _{3}X_{t}T_{t} + \epsilon _{t}$$

where $$Y_{t}$$ was monthly level of adverse outcome, $$T_{t}$$ was the time since the start of the study, $$X_{t}$$ was a dummy representing pre- and post-intervention periods, and $$X_{t}T_{t}$$ was the interaction term, the modified model that tested whether change in contact moderated the values of $$\beta _{2}$$ (change in the level of the outcome) and $$\beta _{3}$$ (difference in slopes of the outcome) would be this:

$$Y_{t} = \beta _{0} + \beta _{1}T_{t} + \beta _{2}X_{t} + \beta _{3}X_{t}T_{t} + \beta _{4}X_{t}C_{t} + \beta _{5}X_{t}C_{t}T_{t} + \epsilon _{t}$$

where $$C_{t}$$ measured monthly amount of contact, $$\beta _{4}$$ tested the difference in pre- and post-intervention periods by average level of contact in the two periods, and $$\beta _{5}$$ tested if the difference in slopes varied by level of contact.

If you wanted, you could define $$C_{t}$$ not as monthly amount of contact but as monthly change in contact (in absolute or relative terms).

• So TtCt would test whether the baseline trend differed between high and low service users? Is it worth including this interaction in the equation? If the principle were extended to a controlled ITS analysis, would the most appropriate equation be: T + X + XT + Z + ZT + ZX + ZXT + TC + XC + XTC + ZC + ZTC + ZXC + ZXTC?
– C_H
Aug 3, 2019 at 12:35
• If you add another term for just $C_t$, the current $\beta_5$ coefficient will then represent the difference between high and low service users in the slope (trend) of the outcome variable after initiation of the intervention compared with pre-intervention (ie, it will act as the difference in differences in slopes). I recommend that you determine how each coefficient is to be interpreted and as a result, whether the model specification aligns with your research question. You can plug sample values for T, C, X, etc. and then modify them one at a time, evaluate and compare results. Aug 4, 2019 at 23:14
• If you have additional observed control variables, they don't necessarily need to be interacted with the current variables (time, intervention dummy). If you don't interact them, you will just be interpreting the main model coefficients as the difference in ___ among individuals who have identical values of the control variables, just like is normally done in multiple regression analysis. Aug 4, 2019 at 23:18