I'm trying to model a dataset in which the goal is to predict the path of users through a series of states. The state space is very simple, with just three states, and I have plenty of sequence observations (~O(10^6)
).
I further know some structure about the transition matrix, as there are only 3 transitions possible:
$$ Y_0 -> Y_0\\ Y_0 -> Y_1\\ Y_0 -> Y_2 $$ and all sequences start in state 0 and end in either state 1 (success) or 2 (failure), so each observed sequence is just a series of $$[Y_0,...Y_0,Y_m] \text{ where } m \in {1,2}$$. Furthermore, there is a fixed maximum sequence length, after which all sequences transition to failure by default.
Where things start to get complicated is that each user has a number of other attributes/features $\mathbf{x} = x_0, …, x_K$, and I would like to include these in the model. I suppose I am looking to learn something like $$ \mathbb{P}[S_{T+1}| S_t,...S_0, X=\mathbf{x}] $$
Put in words, the question I really want to answer is "What is the probability that a user moves to state 1 or state 2, and for how many time steps does it stay in state 0 until then?".
It could be that I'm over-complicating what might look like a simple classification problem, but a key insight we are trying to generate is the second part of the question above:
How long does a user take to convert to success/failure, and how does this depend on their user specific attributes $\mathbf{x}$"
rather than just the simple binary prediction of the end state.
Alternatively, is there anything from survival modelling that might be a better approach to answering this question?