I have time series data from two independent groups. I want to know whether these groups diverge over time and, if so, when they diverge and for how long.
The way I have done this is to estimate (independently for each group) a smoothed spline fit by bootstrap case resampling the data 1000 times and fitting a smoothing spline to each of these resamples. Essentially, this is bootstrap aggregation (bagging) of the spline model fit. Then, I can then get a mean value over time (by averaging the spline model prediction at each point) as well as confidence intervals on this estimate (by taking the standard deviation of the spline model predictions at each point).
Using this resampled dataset and CIs, I can see that the groups sufficiently diverge (p < .05) at timepoint X (e.g., 540ms). But I would like to know what sort of confidence I should have in this estimate. How can I find confidence intervals on this estimate, a single statistic that is the product of bagging?
I have tried resampling the fit splines with replacement, calculating their divergence timepoints, and taking the standard deviation of these estimates as an estimate of standard error. But the estimates using this method are too narrow to be accurate (e.g., 540ms [535,545]), and they get narrower with the number of samples I draw (not an ideal property for a bootstrap) and so I think sampling from the sampled splines is not the way to go.
I have also tried resampling the raw data before sending it into the first procedure, basically creating a two-tiered bootstrap. But here the estimates are extremely wide, and also don't seem correct. Moreover, when I ran some tests using embedded bootstraps on populations with known parameters, this second tier really seemed to mess with estimates of means and standard errors, and so I'm not confident I understand what this second tier of bootstrapping is doing for my estimates.
Anyone have any ideas on how to appropriately estimate the divergence between two bagged smoothing splines?