I am working on a problem where I try to predict the onset of a reaction (of participants), given a set of time-series signals. I know that the event always happened, it is just a matter of question when exactly it happens. The goal is, later on, to predict in real-time whether or when the event would happen, given current (measured) inputs to the model. I see two different ways to approach it: A) as a "classification" problem to predict whether the event would happen or not, at a given time with the current inputs, or B) as a "regression" problem to directly predict the time of the event onset. (I probably did not use those terms correctly in this context but at least to me, those approaches sound like they have the main goal of classification and regression methods, respectively.)

Approach A should output a probability of whether the event would happen or not, and I can think of survival (time-dependent cox), or logistic regression. This approach may, apart from the mean predicted probability, also deliver confidence/prediction intervals of the probability, given current inputs to the model.

Approach B should be done in a probabilistic way to get the full pdf of the predicted onset time, e.g. with a Bayesian regression model (using stan, brms, PyMC3, etc.). Then, by constructing the cdf of the predicted onset time (given current inputs to the model), I could get a similar (?) probability of event onset as in approach A.

My question(s): Which approach sounds favorable for my kind of problem, or could either of them work? Could I compare both with a common evaluation metric like AUC?

My current thinking: I feel like approach B could deliver richer information (the time itself and a probability), and it might be more straightforward to evaluate (e.g., via RMSE). But when looking at previous research, it suggests going for approach A (many medical studies have done it that way, at least).

Edit-1: The starting point is defined as when the participants enter a particular phase of the experiment. All covariates are changing over time and are continuous, except for one that I would like to keep in an interaction, as a boolean variable. All covariates are time-series signals that are not random, rather smooth/slowly changing, continuous, and sampled at 100 Hz.

  • $\begingroup$ Semi-parametric Cox models and fully parametric survival models can provide "the full pdf of the predicted onset time." Please edit your question (comments can get lost) to say more about the details of your study: how is the starting time for each individual defined, are there some covariates you are correcting for that are defined at the starting time for each, and the nature of the "time-series signals" (e.g., how quickly are they varying, are they continuous or categorical, is there some fixed pattern or are they random, do you expect individuals to have a memory of prior signals). $\endgroup$ – EdM Dec 19 '20 at 19:25
  • $\begingroup$ Thanks. I edited the question, adding some details. $\endgroup$ – Tester01 Dec 23 '20 at 19:41

If you want "to predict whether the event would happen or not, at a given time with the current inputs," that's what a survival regression model* does. For example, a Cox regression models the hazard of an event as a function of time and current values of inputs/covariates. With your signal of interest sampled at 100 Hz you might have very large data sets (with time-dependent covariates you need to specify the time span over which each covariate value holds for each individual), but there's no inherent problem (except for memory and computational limits) with handling that in a survival model.

If there might be some memory of past signal levels at work, you would have to define some new covariate that takes the memory into account. The instantaneous covariate values are assumed to determine the instantaneous hazard of an event in such models.

You might consider a fully parametric survival model instead of a Cox model. Then you get a full functional description of hazard as a function of covariate values and time, avoiding difficulties posed by the somewhat awkward empirical baseline hazard function returned by a Cox model. Some parametric models also don't require the (potentially incorrect) proportional hazards assumption of a Cox model, allowing for proportional odds or accelerated failure times instead. If your inputs have well defined functional forms over time, a parametric model might also simplify the data-handling and modeling process.

I don't see that your Approach B is much different from what a parametric survival model would do, except for approaching the problem from a Bayesian rather than a frequentist perspective.

One warning: there's a risk of survivorship bias with time-varying covariates. For example, if your "smooth/slowly changing, continuous" input signal level changes systematically over time, you might have difficulty separating out the characteristics of the signal from the influence of time per se, and those who just happen to hang on longer before the event will be exposed to different signal levels. To work that through, you will have to apply your knowledge both of the subject matter and of your experimental design.

*As you acknowledge, the regression/classification distinction you try to make isn't quite standard usage. For example, logistic regression can be used for classification once you choose a probability cutoff for class assignment. A survival regression model explicitly takes elapsed time into account along with instantaneous covariate values.

  • $\begingroup$ Thank you very much for the detailed reply, and sorry for the late follow-up. It sounds very reasonable to try parametric survival models. Most implementations of time-dependent survival models I found so far were using the Cox proportional hazards approach, but I have come across some packages that implemented time-dependent AFT models. I will also investigate the survivorship-bias issue you mentioned. $\endgroup$ – Tester01 Jan 29 at 17:55
  • $\begingroup$ The difference I see with the Bayesian GLM (approach B) is that I would use much less data for training the model. I would only use the covariate values at the time of event, as opposed to the whole time sequence of data used for the survival approach. So I would suspect that it might underperform compared to the survival approach? That is why I wondered if I could compare both methods with a classification performance metric like AUC, over time. $\endgroup$ – Tester01 Jan 29 at 17:58
  • $\begingroup$ @Tester01 frequentist survival models also use only covariate values at the time of an event (meaning: values for all individuals at risk at any event time), so you could format the data accordingly. Also, unless you have thousands of participants, you probably shouldn't be setting aside separate training and test sets as you lose precision both in modeling and in testing. See for example Harrell's blog post on the topic and Section 5.3 of his course notes. Bootstrap validation is typically much better. $\endgroup$ – EdM Jan 29 at 18:42
  • $\begingroup$ Thanks for clarifying, and thanks for the examples. But, if I use all samples of my data, I will have much more samples that do not include the event, and only few that include it (at the very end of each trial for a participant). I tried fitting a parametric survival model, but the survival probability is very low at all times, as expected. Am I missing anything? $\endgroup$ – Tester01 Feb 9 at 17:45
  • $\begingroup$ @Tester01 you said that "I know that the event always happened, it is just a matter of question when exactly it happens." So just having lots of data points representing times without events isn't really a problem; that's what you'd expect in a situation with covariates that vary often over time. The biggest question I see is whether the current covariate values are related to the hazard of the event, or if you have to take into account some measure of the history. $\endgroup$ – EdM Feb 9 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.