1
$\begingroup$

In my problem I am trying to predict the time from a vehicle leaves a charging station until the next one arrives using various techniques, of which Cox Proportional Hazard and DeepSurv are some of them. When training a model I do the following:

  1. Decide upon a set of hyperparameters for tuning.
  2. Split data in training and test data.
  3. Loop through every combination of hyperparameters.
    1. Perform K-fold cross-validation on the training set.
    2. Choose the hyperparameter combination with the best results.
  4. Train a model with the best-found parameters on all training data.
  5. Evaluate the test set with the model from the previous point.

In 3.2. I am however uncertain about how to evaluate the model. A classic approach would be to choose to use the C-index. From my problem perspective, this might not be so useful though, as the C-index is the model's ability to correctly provide a reliable ranking of the survival times (taken from pysurvival). What is more interesting is the actual predictions which for example can be shown by the RMSE which by taking the AUC of the survival function can be estimated.

Is this the right way to think about it or am I on the wrong path?

$\endgroup$
9
  • $\begingroup$ So am not too familiar with cox, but would suggest against it (as its aim is to infer effects, ignoring the baseline survival). Can you discretise the time? in that case you could just use a logistic regression. $\endgroup$
    – seanv507
    Commented Apr 21, 2021 at 16:52
  • 1
    $\begingroup$ However, survival analysis seems to be completely the wrong approach to predict time between arrival times. Could you clarify what you think drives arrival times... eg is there no queuing $\endgroup$
    – seanv507
    Commented Apr 21, 2021 at 17:00
  • 1
    $\begingroup$ I'm not so pessimistic as @seanv507 about using survival analysis, as one might model this as a set of recurring events.See the main and multi-state vignettes of the R survival package. Don't know if that's how you're modeling, how you are incorporating covariates into the model, etc. Not clear what "discrimination" as measured by C-index would represent in that case. For your application a measure of calibration is probably more important. A parametric model, maybe Poisson based, would probably be more useful than Cox. $\endgroup$
    – EdM
    Commented Apr 21, 2021 at 19:28
  • $\begingroup$ @seanv507 so my question is more of a research question to also test how and if survival analysis can model the problem. I do not see how a logistic regression would be beneficial as I am trying to predict the time (so for example minutes) between a vehicle leaving and another arriving. It is a very hard problem with few features due to no knowledge about users and as the stations obviously are stationary. At the moment my features are time and knowledge about previous behaviour (not exactly time series alike, but similar). $\endgroup$
    – andKaae
    Commented Apr 22, 2021 at 7:45
  • $\begingroup$ @EdM what do you excatly mean with a measure of calibration? And why do you think a Poisson based model would be more beneficial? $\endgroup$
    – andKaae
    Commented Apr 22, 2021 at 7:50

1 Answer 1

2
$\begingroup$

To answer the title of the question, as survival models are fit by maximizing likelihood (partial likelihood for a Cox model), that's the best cross-validation measure for survival analysis: the (partial) likelihood of the held-out data given the parameter estimates. Even if the C-index is appropriate in your application, Frank Harrell (who originated the C-index for survival models) doesn't find it adequately sensitive to distinguish between competing models. See this thread, for example. I don't use Python so I can't help with implementation directly; packages for survival analysis in Python should show the way.

For evaluating the final model, what can be most useful is whether its event-time probability predictions are close to those observed. That's what I meant by "calibration" in a comment. For example, order the events in your test set in terms of the model-predicted probability of having an "event" before a specified time, group the events into bins accordingly, and plot the observed proportion of events before that time within each bin against the predicted probability. Repeat for a set of times of interest. There are ways to do this more smoothly, without binning, for standard single-event data but I haven't thought through whether they would apply here. If the model is well calibrated, any other measure of interest (e.g., RMSE) should be as good as you are going to get, so evaluate the probability calibration first.

Is survival modeling appropriate here?

Discussion in comments had more to do with the last sentence of the question:

Is this the right way to think about it or am I on the wrong path?

That led to multiple comments about whether and how survival analysis is appropriate for this application. The terminology can hide a lot of commonalities. For example: there are close relations between survival models of the type appropriate here and other models of point processes (e.g., Poisson models) or multi-state processes; discrete-time survival analysis is essentially a set of logistic regressions; Cox models and fully parametric models have many similar strengths and weaknesses. So the issue is which form of multi-state/repeated event analysis works best here.

There are two general issues to consider. First, how are you defining time = 0 for survival analysis? Should that be the initial installation time of the charging station? The time you started data collection? Should time reset to 0 for a particular station each time it becomes available? Second, you can't assume that all "events" are independent; within individual stations, events and their associations with covariates are likely to be correlated. Survival software referenced below handles that; if you are writing your own, make sure that you do, too.

Beyond those general considerations, the question is what type of survival model to use. This might be structured simply as a repeated-event survival model, but you need to apply your knowledge of the subject matter to decide. A recurrent-event survival model assumes "a single event type, with the possibility of multiple events per subject" (Section 2.2 of the R survival vignette). An implicit assumption in a simple repeating-event model is that the "event" is effectively instantaneous. Does that make sense here? Is the duration of occupancy of a charging station that much shorter than the time between vehicle arrivals?

You thus might need to move at least to a two-state model, modeling transitions both from occupied to unoccupied and vice-versa (perhaps modeling things like "out-of-order" states, too). The R multi-state vignette provides a useful introduction.

Another question is whether to use a Cox-type semi-parametric model or a fully parametric model. Cox models have the advantage/disadvantage of not specifying a parametric form for the baseline hazard. They can be used for predictions, provided that the baseline hazard estimated from the model* is representative of the population of interest. You do need to document that the proportional hazards assumption holds adequately.

Fully parametric models have advantages in terms of providing equations for describing the model and making predictions. They aren't necessarily better, however.

With a parametric model, you must document that the chosen parametric family models the data adequately, like you have to document proportionality of hazards in a Cox model. For generalization, you must assume that the chosen form will also apply to new samples of the population of interest, like you have to assume that the empirical baseline hazard of a Cox model will adequately represent new samples. Assumptions relating to proper model specification (e.g., linearity in predictors) ultimately are the same with standard regression models of either type, or, for that matter, with discrete-time models.

The R flexsurv package can handle multi-state parametric models. Its multi-state modeling vignette is a good resource for helping to think through how to formulate a multi-state model even if you aren't going to use the package itself and will use boosted trees or neural nets instead.


*A Cox model does not assume a particular baseline hazard; it just assumes that covariate-associated differences in hazard are proportional, whatever the baseline hazard function might be. If that assumption holds, then the Cox regression-coefficient estimates plus the original data on covariate values and survival directly provide an empirical baseline hazard.

$\endgroup$
1
  • 1
    $\begingroup$ My point was that if you are trying to find the best time estimate you want to explicitly fit/smooth the baseline hazard $\endgroup$
    – seanv507
    Commented Apr 22, 2021 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.