How do you include change in a variable over time across participants within a GAMM framework? I am really struggling with trying to understand how to specify a GAMM model that predicts Y from the changes in two variables X1 and X2 over time across multiple participants over a varying amount of time.
This is what my data looks like:
| Y   | X1  | X2  | time | ID |
|-----|-----|-----|------|----|
| 1.7 | .50 | .77 | 1    | 1  |
| 1.7 | .45 | .79 | 2    | 1  |
| 1.7 | .30 | .83 | 3    | 1  |
| 1.7 | .35 | .72 | 4    | 1  |
| 2.4 | .51 | .29 | 1    | 2  |
| 2.4 | .52 | .23 | 2    | 2  |
| 2.4 | .50 | .37 | 3    | 2  |
| 2.4 | .45 | .39 | 4    | 2  |
| 2.4 | .49 | .33 | 5    | 2  |
| 0.6 | .60 | .17 | 1    | 3  |
| 0.6 | .65 | .29 | 2    | 3  |
| 0.6 | .70 | .13 | 3    | 3  |
| 0.6 | .60 | .27 | 4    | 3  |
| 0.6 | .65 | .19 | 5    | 3  |
| 0.6 | .60 | .23 | 6    | 3  |

Key features of this dataset:


*

*Time varies from participant to participant, to the point where some sequences are 10x as long as others.

*There are over 900 participants, but even so, almost every participant has a unique Y value. This means when I add a random effect for ID, it explains nearly all the variance in Y.

*X1 and X2 range between 0 and 1 are on the the same scale. They covary, but not perfectly.
Main Questions:


*

*Is this even possible to model using a GAMM? Should I be thinking about something else?

*I am not so interested in using time as a global predictor in and of itself, but rather using the changes in X1 and X2 over time to predict Y. How would I model this? Random slopes?

*How would I take into account the varying time lengths? Do I need to somehow standardize all of the lengths during preprocessing?

*Could I model a random effect of ID somehow? It seems misleading to run the analysis without including that important source of variation.
I am using the mgcv in R, it would be incredibly helpful if some kind soul would help answer the above questions in a way that would help me know what type of syntax I would be looking to incorporate. I have spent a couple weeks reading up on anything I could to try to answer this question, and here is the model that I was able to construct so far. Am I on the right track?
library(mgcv)

fit <- bam(Y ~ 

           s(X1) +                    #Nonlinear main effect of X1
           s(X2) +                    #Nonlinear main effect of X2
           s(X1, time, bs = "fs") +   #Nonlinear interaction of X1 over as.factor(time)
           s(X2, time, bs = "fs") +   #Nonlinear interact-on of X2 over as.factor(time)
           s(time, ID, bs = 're'),    #Random slope for time by as.factor(ID) ???

       data = df, family = "gaussian", method = "fREML",
       discrete = TRUE, nthreads = detectCores()-1)

summary(fit)
plot(fit, pages = 1, all.terms = TRUE, shade = TRUE)

 A: First off, including two variables that covary (the "not perfectly" gives me concern as that implies the correlation is high, just not 1) is going to be problematic, but if we ignore that for now...
The answers below are making an assumption that time is common to all IDs (like a date in the real world) or is a relative measure, such as time from some important event.
Q1
Yes, you could do this with a GAMM. You'd be assuming that the effects of X1 and X2` on the response are nonlinear.
Q2
While you are not interested in time per se, you should probably include a s(time) to model change in the response over time that is not captured by the time-varying values of X1 and X2.
As X1 and X2 change over time — their values are not constant — then they should explain (or might assume that they would) some of the variation in the response over time. Simply by including linear or smooth effects of these two covariates you are accounting for the time-structured effects of these variables.
If you expect the effect of X1 or X2 on the response to change over time — i.e. for the same value of X1 or X2 the effect on the response is different at time $t$ and $t_{+s}$ &mdash, then you would want a tensor product of time and the covariate. For example, including
te(X1, time)

in the model would allow the smooth effect of X1 to vary as a smooth function of time.
Q3
Given my assumptions about the time variable mentioned above, you wouldn't need to do anything special in the model to account for the different amounts of time in each series. You would want to be careful not to predict outside the range of each series (or not predict too far outside each range) as the uncertainties will increase considerably.
You might want to include a global (trying to avoid the term "population") smooth for time which tries to capture how all series change on average over all time points. That would give you some basis for predicting beyond the range of time in the shorter series. To do that you'd include:
s(time, k = K)

in the model, setting K to be some suitable value for the sort of general trend you might expect (plus a bit).
Q3
Yes, you will need to include something to account for the between ID variation in the response. You might also want to include random smooths of time by ID to account for time-structured effects in each series that are not captured by the global smooth of time.
You might also want to have ID specific effects of X1 and X2 if you think that these effects vary between IDs.

A simple model to start with would be:
Y ~ s(time) +          # common time-structured effects not modelled by...
    s(X1) +            # smooth effect of X1
    s(X2) +            # smooth effect of X2
    s(ID, bs = 're')   # random intercepts for factor ID

A more complex model that allowed for random smooths of X1 and X2 would be
Y ~ s(time) +              # common time-structured effects not modelled by...
    s(X1, ID, bs = 'fs') + # random smooth effect of X1
    s(X2, ID, bs = 'fs') + # random smooth effect of X2
    s(ID, bs = 're')   # random intercepts for factor ID

If there is unmodelled time effects remaining within IDs then you could add random smooth trends using the fs smooths also:
Y ~ s(time) +                # common time effects not modelled by...
    s(X1, ID, bs = 'fs') +   # random smooth effect of X1
    s(X2, ID, bs = 'fs') +   # random smooth effect of X2
    s(time, ID, bs = 'fs') + # random smooth time effects
    s(ID, bs = 're')         # random intercepts for factor ID

If you want a time-varying, random smooth effects of X1 and X2, you are looking at quite a complex model:
Y ~ s(time) +
    t2(X1, time, ID, bs = c('tp','tp','re'), full = TRUE) +
    t2(X2, time, ID, bs = c('tp','tp','re'), full = TRUE) +
    s(time, ID, bs = 'fs')
    s(ID, bs = 're')

You are probably going to want to use method = REML (or method = 'fREML' (for bam()) and also use select = TRUE for the very complex models and if X1 and X2 do covary so much as this setup should be most robust to fitting problems with concurved effects.
If you give more details about time and what it is I can revisit some of what I wrote here to see if remains valid.
