For a predictive model (e.g., like a regression or mixed model), what statistical methods are best used when one wants to model data where a variable number of observations occur between some initial value (pre) and some final value (post)? To note, in the cases I encounter, it is not traditional unbalanced panel data (e.g., everyone on same time points, but some missing) but instead usually uneven due to subject-controlled processes (e.g., subject might do more tasks in an hour of using a system) or system paths (e.g., a choice made by a user might lead to earlier completion of a session). In terms of structure, you might think about each subject $i$ as having a timeline of observations in a form similar to panel data, but where spacing between events for different subjects might not be exactly the same (though either spaced between a similar order of magnitude, or with exact timestamps known):
Pre-Score: Variable $O_{i,Pre}$
- Event 1: Variables $A_{i,e=1}$, $B_{i,e=1}$, ...
- Event 2: Variables $A_{i,e=2}$, $B_{i,e=2}$, ...
- ...
Post-Score: Variable $O_{i,Post}$
Where we can assume for the sake of argument that all main variables of interest are either binary or continuous (e.g., no categorical variables, except for things that would be considered as a condition for random effects like subject id).
This is a fairly common issue that I have run across, in different data sets. To summarize, you have the outcome variable you really care about in the pre/post. You have a bunch of other things that you get multiple observations for, where the number of observations varies and there are multiple metrics you know about each observation. Examples would be metrics about learning activities (e.g., time, # clicks) in between two tests, metrics about speeches that a leader makes between an electoral promise versus their vote on the issue, browser history of web pages in between assessing familiarity with a topic, etc.
If the number of events and variables is the same for each subject, this can be handled a few ways. These include:
- Averaging (or otherwise aggregating, such as through time series metrics) each of the variables across events: $\overline{A_i} =\sum_{j}^N A_{i,j}/N$, to enable $O_{i,Post} ~ O_{i,Pre} + \overline{A_i} + ...$
- Flattening all observations into one big model, as per: $O_{i,Post} ~ O_{i,Pre} + A_{i,e=1} + B_{i,e=1} + ... + A_{i,e=2} + B_{i,e=2} + ... $
- Reducing the number of variables for each event (e.g., via Principle Component Analysis, theoretical groupings) and then using the smaller model in a similar form as the flattened model above.
- Auto-regressive modeling, typically with a hidden variable or variables that generate both the observations of $O$ and also the variables that occur during an event ($A$, ...), even though both are never observable at the same time, by relying on their temporal relationships (e.g., that $A_{i,e=1}$ should be more closely related to $O_{i,Pre}$ than $A_{i,e=100}$ would be). By intuition, this would seem to be much more trustworthy if you had more observations of $O$ than just a pre and post though. Going down this road seems to lead to either machine learning (e.g., Hidden Markov Model variant) or custom programming solution at the problem, so you can potentially get a predictive model, but with little ability to interpret it.
Worse, these solutions often suffer once you have a variable number of events per subject. Averaging loses temporal relationships and the power of more observations. Time series analysis for generating aggregates doesn't make sense if events are ordered but not on a timeline. I've used subsequence analyses like n-tuple frequencies for ordered categorical data, but it's a pretty heavy step for pre-processing and doesn't translate readily to continuous variables. Flattening, with or without something like PCA, implies that either some subject's events are truncated or others are padded with some form of N/A's. A shared latent variable approach could theoretically work, but if it was a nightmare to train and interpret with a similar set of events, this appears to be even worse.
So then, what statistical techniques have people found that helped them produce classically-interpretable statistical results for this type of data? The goal is to get to a model form where it is possible to report commonly-understood results such as magnitude (e.g., coefficients, transition probabilities), strength of evidence (e.g., p values, test statistics), and bounded measures of model fit (e.g., $R^2$, added variance explained)? While this problem formulation may indeed have no general model that can be readily interpreted and reported, I am interested in knowing what classes of approaches have been effective for data in this form, even if they have boundary conditions or assumptions about the structure of the data.
