I am not from statistics background but I am faced with data that seems to be survival data. First of all, I read about survival analysis and I know about recurrent survival data and different models (AG, PWP, Frailty, WLW) for it. Data description The data that I have is from clinical trials: Each patient was supposed to have a specific test (T) each year and from this test we can "guess" the percentage of infection (POI). The data also has more covariates about patients; Age, Sex, DOB, Race, and more. All in all, some patients did all T tests in 10 years (Folow-up time) while the others did some of these tests. The goal To predict the POI for next visit from previous or current test. I already restructured the data such that I can use it with one of previous recurrent models. Suggestion I read about Recurrent Neural Network and I am thinking to use it as a predicting mechanism, is this possible?
I read about Recurrent Neural Network and I am thinking to use it as a predicting mechanism, is this possible?
I am less familiar with survival analysis, but this seems to be a fine fit for an RNN. You define $T$ as the test but also as the number of tests, but let's go with the second one and say that for each patient you have $T$ tests. The RNN would take as input the lengh-$T$ sequence of $K$-dimensional vectors (assuming you have $K$ features, which include the covariates and the test) and output the POI. So you would be mapping from a sequence to a scalar.
Survival analysis generally seems to be concerned with 'time-to-event' estimation (eg, time to death), so might not be as applicable here, where you're not concerned with time, although you might run into issues like censoring (eg, missing data because the patient didn't follow up) associated with survival analysis.
Depending on the amount of data, an RNN might have too much capacity for this task, but it wouldn't hurt to try. You could also try simply using fixed-length representations for the sequences and using a non-temporal model. For example, transforming each patient's sequence by taking their mean previous POI and seeing if it predicts the next POI as features in a regression.
My point of interest is to predict the POI for next visit from previous or current test.
The benefit of the RNN would be that you could predict POI of the next visit taking into account all previous tests, covariates from a patient---ie, the RNN will remember not just the most recent but all tests.
How do you deal with misaligned, irregularly sampled, missing clinical data?
Lipton, Zachary C., David C. Kale, and Randall Wetzel. "Modeling missing data in clinical time series with rnns." Machine Learning for Healthcare (2016).
Futoma, Joseph, et al. "An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection." arXiv preprint arXiv:1708.05894 (2017).