How to test whether 2 prediction intervals are statistically different?

Question

I've been struggling with this for a while now, hopefully someone will know how to help me :)

Here it is :

1) I'm using a linear mixed effects model on longitudinal data (biological values of many patients over time). Basically, the model predicts a quatitative outcome from the treatment group (2 groups) and the time since treatment initiation.

2) Note that i force the model to deny any baseline difference between groups, by removing the treatment effect by itself => the treatment effect is thus only present as an interaction with time. Apparently this is legit in my case, because there is no reason for the treatment to impact baseline value since it has "just started".

3) Anyways, now i want to predict an "average" trajectory for each treatment group : i thus simulate the treatment by "fixing" the treatment group for every point in time, and i obtain a prediction for each group at every time mark. If i plot this, i obtain to curves (lines) with similar intercept but different slopes.

4) Now i've only performed the simulation once for each treatment at each point in time. If i repeat the process 1000 times, I obtain a distribution of predictions for each group at every timemark. I extract the 2.5 and 97.5 percentiles of my predictions, which gives me a 95% prediction interval, for each group at every point in time.

5) I have one particular time mark that I'm interested in : i thus have 2 prediction means with prediction intervals (one for each treatment group).

Here's the question : how do I compare those 2 prediction intervals ? Regular test use the standard error, which is linked to the number of observations, but in my case observations are just simulations. What i mean is I could easily Perform 1 million simulations and thus obtain a somewhat "artificial" statistical difference.

I'm pretty sure I got something wrong somewhere, just not sure where ...

Thanks in advance for your help !

Answer 1

From 1), the model can be written as:

$$Y=\beta_0 +\beta_1 T +\beta_2 TX + \text{ random effects } + \epsilon$$ where $T$ is time and $X=0$ for treatment 1, and = 1 for treatment 2. You used this model means you think you data meet the assumptions of this model.

For 2), if the time is the time from baseline date (it means for baseline the time =0), your method is correct. If not, (for example age is used) the treatment effect should be kept.

3) and 4): It is unnecessary to do simulation to get the statistical inference under the specified mixed linear model, because under the assumptions of this model, the distribution of the estimates are well established. If you want to predict the mean of $Y$ for treatment $x$ at time $t$ then $\hat{\mathrm{E}(Y|T=t,X=x)} = \hat\beta_0 +\hat\beta_1 t +\hat\beta_2 tx$. Its confidence interval can be constructed based on $t$ distribution with given estimated variance and degree of freedom.

5) For testing the difference between treatment at the specified time $t$, you still can construct the linear combination of the parameters. For treatment 1, $$\mathrm{ E}(Y) = \beta_0 +\beta_1 t$$ For treatment 2, $$\mathrm{ E}(Y) = \beta_0 + \beta_1t + \beta_2t$$
Their difference is $\beta_2t$. So by testing the null hypothesis $H_o: \beta_2t = 0$, you get the result. In fact, its p-value is equal to p-value given at the estimate of $\beta_2$ by most statistical software. It is easy to get the its confident interval.