How can machine learning models (GBM, NN etc.) be used for survival analysis?

Question

I know that traditional statistical models like Cox Proportional Hazards regression & some Kaplan-Meier models can be used to predict days till next occurrence of an event say failure etc. i.e Survival analysis

Questions

1. How can regression version of machine learning models like GBM, Neural networks etc be used to predict days till occurrence of an event?
2. I believe just using days till occurence as target variable and simplying running a regression model will not work? Why wont it work & how can it be fixed?
3. Can we convert the survival analysis problem to a classification and then obtain survival probabilities? If then how to create the binary target variable?
4. What is the pros & cons of machine learning approach vs Cox Proportional Hazards regression & Kaplan-Meier models etc?

Imagine sample input data is of the below format

Note:

• The sensor pings the data at intervals of 10 mins but at times data can be missing due to network issue etc as represented by the row with NA.
• var1,var2,var3 are the predictors, explanatory variables.
• failure_flag tells whether the machine failed or not.
• We have last 6 months data at every 10 min interval for each of the machine ids

EDIT:

Expected output prediction should be in the below format

Note: I want to predict the probability of failure for each of the machines for the next 30 days at daily level.

Answer 1

For the case of neural networks, this is a promising approach: WTTE-RNN - Less hacky churn prediction.

The essence of this method is to use a Recurrent Neural Network to predict parameters of a Weibull distribution at each time-step and optimize the network using a loss function that takes censoring into account.

The author also released his implementation on Github.

Answer 2

Have a look at these references:

https://www.stats.ox.ac.uk/pub/bdr/NNSM.pdf

http://pcwww.liv.ac.uk/~afgt/eleuteri_lyon07.pdf

Also note that traditional hazards-based models like Cox Proportional Hazards (CPH) are not designed to predict time-to-event, but rather to infer variables' impact (correlation) against i) observations of events and, hence ii) a survival curve. Why? Look at the CPH's MLE.

Hence, if you want to more directly predict something like "days till occurrence", CPH may not be advisable; other models may better serve your task as noted in the above two references.

Answer 3

As @dsaxton said, you can build a discrete time model. You set it up to predict p(fail at this day given survived up to previous day). Your inputs are current day (in whatever representation you want) eg one hot encoding, integer,.. Spline... As well as any other independent variables you might want

So you create rows of data, for each sample that survived till time t-1, did it die at time t (0/1).

So now the probability of surviving up to time T is the product of p(don't die at time t given didn't die at t-1) for t=1 to T. Ie you make T predictions from your model and then multiply together.

I would say the reason its not such an idea to directly predict time to failure is because of the hidden structure of the problem. Eg what do you input for machines that didn't fail. The underlying structure is effectively the independent events: fail at time t given didn't fail up to t-1. So eg if you assume it is constant, then your survival curve becomes an exponential (see hazard models)

Note in you case you could model at 10 minute interval or aggregate up the classification problem up to day level..