To answer the title of the question, as survival models are fit by maximizing likelihood (partial likelihood for a Cox model), that's the best cross-validation measure for survival analysis: the (partial) likelihood of the held-out data given the parameter estimates. Even if the C-index is appropriate in your application, Frank Harrell (who originated the C-index for survival models) doesn't find it adequately sensitive to distinguish between competing models. See this thread, for example. I don't use Python so I can't help with implementation directly; packages for survival analysis in Python should show the way.
For evaluating the final model, what can be most useful is whether its event-time probability predictions are close to those observed. That's what I meant by "calibration" in a comment. For example, order the events in your test set in terms of the model-predicted probability of having an "event" before a specified time, group the events into bins accordingly, and plot the observed proportion of events before that time within each bin against the predicted probability. Repeat for a set of times of interest. There are ways to do this more smoothly, without binning, for standard single-event data but I haven't thought through whether they would apply here. If the model is well calibrated, any other measure of interest (e.g., RMSE) should be as good as you are going to get, so evaluate the probability calibration first.
Is survival modeling appropriate here?
Discussion in comments had more to do with the last sentence of the question:
Is this the right way to think about it or am I on the wrong path?
That led to multiple comments about whether and how survival analysis is appropriate for this application. The terminology can hide a lot of commonalities. For example: there are close relations between survival models of the type appropriate here and other models of point processes (e.g., Poisson models) or multi-state processes; discrete-time survival analysis is essentially a set of logistic regressions; Cox models and fully parametric models have many similar strengths and weaknesses. So the issue is which form of multi-state/repeated event analysis works best here.
There are two general issues to consider. First, how are you defining
time = 0 for survival analysis? Should that be the initial installation time of the charging station? The time you started data collection? Should
time reset to
0 for a particular station each time it becomes available? Second, you can't assume that all "events" are independent; within individual stations, events and their associations with covariates are likely to be correlated. Survival software referenced below handles that; if you are writing your own, make sure that you do, too.
Beyond those general considerations, the question is what type of survival model to use. This might be structured simply as a repeated-event survival model, but you need to apply your knowledge of the subject matter to decide. A recurrent-event survival model assumes "a single event type, with the possibility of multiple events per subject" (Section 2.2 of the R
survival vignette). An implicit assumption in a simple repeating-event model is that the "event" is effectively instantaneous. Does that make sense here? Is the duration of occupancy of a charging station that much shorter than the time between vehicle arrivals?
You thus might need to move at least to a two-state model, modeling transitions both from occupied to unoccupied and vice-versa (perhaps modeling things like "out-of-order" states, too). The R
multi-state vignette provides a useful introduction.
Another question is whether to use a Cox-type semi-parametric model or a fully parametric model. Cox models have the advantage/disadvantage of not specifying a parametric form for the baseline hazard. They can be used for predictions, provided that the baseline hazard estimated from the model* is representative of the population of interest. You do need to document that the proportional hazards assumption holds adequately.
Fully parametric models have advantages in terms of providing equations for describing the model and making predictions. They aren't necessarily better, however.
With a parametric model, you must document that the chosen parametric family models the data adequately, like you have to document proportionality of hazards in a Cox model. For generalization, you must assume that the chosen form will also apply to new samples of the population of interest, like you have to assume that the empirical baseline hazard of a Cox model will adequately represent new samples. Assumptions relating to proper model specification (e.g., linearity in predictors) ultimately are the same with standard regression models of either type, or, for that matter, with discrete-time models.
flexsurv package can handle multi-state parametric models. Its multi-state modeling vignette is a good resource for helping to think through how to formulate a multi-state model even if you aren't going to use the package itself and will use boosted trees or neural nets instead.
*A Cox model does not assume a particular baseline hazard; it just assumes that covariate-associated differences in hazard are proportional, whatever the baseline hazard function might be. If that assumption holds, then the Cox regression-coefficient estimates plus the original data on covariate values and survival directly provide an empirical baseline hazard.