I am not from statistics background but I am faced with data that seems to be survival data. First of all, I read about survival analysis and I know about recurrent survival data and different models (AG, PWP, Frailty, WLW) for it. Data description The data that I have is from clinical trials: Each patient was supposed to have a specific test (T) each year and from this test we can "guess" the percentage of infection (POI). The data also has more covariates about patients; Age, Sex, DOB, Race, and more. All in all, some patients did all T tests in 10 years (Folow-up time) while the others did some of these tests. The goal To predict the POI for next visit from previous or current test. I already restructured the data such that I can use it with one of previous recurrent models. Suggestion I read about Recurrent Neural Network and I am thinking to use it as a predicting mechanism, is this possible?


I read about Recurrent Neural Network and I am thinking to use it as a predicting mechanism, is this possible?

I am less familiar with survival analysis, but this seems to be a fine fit for an RNN. You define $T$ as the test but also as the number of tests, but let's go with the second one and say that for each patient you have $T$ tests. The RNN would take as input the lengh-$T$ sequence of $K$-dimensional vectors (assuming you have $K$ features, which include the covariates and the test) and output the POI. So you would be mapping from a sequence to a scalar.

Survival analysis generally seems to be concerned with 'time-to-event' estimation (eg, time to death), so might not be as applicable here, where you're not concerned with time, although you might run into issues like censoring (eg, missing data because the patient didn't follow up) associated with survival analysis.

Depending on the amount of data, an RNN might have too much capacity for this task, but it wouldn't hurt to try. You could also try simply using fixed-length representations for the sequences and using a non-temporal model. For example, transforming each patient's sequence by taking their mean previous POI and seeing if it predicts the next POI as features in a regression.

My point of interest is to predict the POI for next visit from previous or current test.

The benefit of the RNN would be that you could predict POI of the next visit taking into account all previous tests, covariates from a patient---ie, the RNN will remember not just the most recent but all tests.

How do you deal with misaligned, irregularly sampled, missing clinical data?

  • See:

    Lipton, Zachary C., David C. Kale, and Randall Wetzel. "Modeling missing data in clinical time series with rnns." Machine Learning for Healthcare (2016).

    Futoma, Joseph, et al. "An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection." arXiv preprint arXiv:1708.05894 (2017).

  • $\begingroup$ I really appreciate your FAST respond to my question and the valuable info. that you mention. I will go with your suggestions and let you know what is the results. Thanks user99889 $\endgroup$
    – Abdal
    Sep 3 '17 at 17:16
  • $\begingroup$ So I read about RNN and to fit this to data I am confused if what I am thinking is right. Let say I have five patients A,B,C,D,E with 10,3,7,1,2 T test respectively. What about the missing data, i.e patient A has a length 10 vector of let say 7 features while the B patient will be with 3 length and so on? what about the value of each test? Please more clarification and details. $\endgroup$
    – Abdal
    Sep 10 '17 at 16:50
  • $\begingroup$ Missing data in clinical time series is a huge issue that would have to be dealt with using any time series model, although it's much better to deal with it as you describe than to use a time window approach and deal with it implicitly, sometimes unconsciously. A naive approach: carry the most recent value over (forward fill). A better approach: use a statistical model to interpolate. Also see my edits. $\endgroup$
    – user0
    Sep 10 '17 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.