5
$\begingroup$

I'm looking for a principled way to estimate when an event of interest is overdue (a binary decision/alert), not just predicting when it is supposed to happen.

In the survival analysis literature I have not come across this particular question yet: the typical survival paper/algorithm is about estimating 'how much longer' until an event occurs -- which is never going to be overdue, because by construction it's always in the future. While this makes sense mathematically/probabilistically/conceptually, it's not helping to define formally what it means for an event to be overdue and how one could alert on this based on a probabilistic framework.

As a concrete example, consider the problem of alerting doctors when the patient's recovery process goes 'off track', i.e., alert them when recovery is overdue (vs potential complications looming). A doctor at "t=0" (day of surgery) has their own predictive distribution for recovery time based on personal experience, and tells patient that they are going to check in again at some point in the future (say, the 80% quantile). Yet, if patient has a heart rate & body temperature monitor on (time-varying features), then every moment a predictive model could tell us the conditional distribution of recovery given heart rate & body temperature. This conditional distribution is always in the future and it moves further into the future with every day passing by some $\Delta > 0$ (conditional survival curves).

Considerations so far:

  • take the prediction interval from a t=0, and alert when observed duration is outside the interval: that works assuming as long as I don't have access to time-varying features. In doctor example, if I have a data feed on "blood pressure", "cholesterol", and "body temperature", then ideally I want to take this information into account for alerting, not rely on information at t=0. But I don't just want to rely on doctor telling me "when cholesterol gets higher than 'X' we should check in again" -- rather rely on models to forecast that recovery pattern [and augment doctor's opinion].

  • take prediction interval from 'yesterday': that predictive distribution [using 'blood pressure' and 'cholesterol' features], will in practice have the predictive distribution far into the future -- no matter how long we already go without recovery -- with a $Pr(\text{time until recovery} \leq 24hrs) \approx 0$.

  • don't predict remaining time, but always predict "total time": that solves the problem as now 'overdue' is just when total prediction is less than observed total. However, not a fan of this as I have a model predicting already (partially) observed events (+ inconsistencies when using a total duration model make statements about future duration).

  • re-frame as 'determine best snapshot to use as baseline' problem: find the best snapshot s, 0 < s < t (today), for which it makes sense to consider features up to "s", and then use the prediction interval from $Pr_s(\text{time until recovery} \mid \text{features at day s}, \text{survived s days})$ and compare it to actual observed duration since "s", t - s. Alert when that probability is above a certain cutoff (say, 90%). That does not really solve the problem, but just moves it out of the traditional survival analysis framework.
    One idea here is to use that snapshot 's', where the conditional survival curve based on forecast at time 's' given it has survived until today is close to the current predictive distribution using all features up to time 't'. I.e. start today at 't' and move it back as far as possible as long as the distance (KL divergence?) between $Pr_t(\text{remaining duration} \mid \text{features at time t})$ and $Pr_s(\text{ remaining duration} \mid \text{ features at day s, survived another t-s days})$ is "small". Determine what is a good "small" threshold by backtesting and using empirical null hypothesis estimation (Efron). Then use the prediction interval from this optimal s*, compare to observed duration (t - s*), and alert if outside interval.

  • Treat as a causal inference problem, where "alert" is the treatment variable, duration until recovery is the outcome variable, and now it's a classic causal inference problem to predict unit level treatment effect (UTE) and pick the one where it has highest impact in reducing duration (to recovery) -- e.g., Kelly, Kong, and Goerg (2022) for a UTE causal ML algorithm. Issue is that we don't have historically observed this 'alert' as a treatment; alerting is a new concept that we haven't measured historically.

What I want to provide the doctor/hospital is a statistical/ML system to page the doctor when patient is 'overdue' that takes time-varying covariates into account but does not trivially never alert, because based on current model predictions recovery is always due (far) in the future. Looking for any references to existing work (maybe I'm just searching for the wrong keywords/terminology to find papers that talk about this problem) or pragmatic ideas that can be implemented using estimated survival predictive distributions (e.g., from Weibull RNNs).

Update 2022/10/31: I removed the truck maintenance example from original post. My use case is the doctor example, where it's of interest to make the event happen quickly -- and catch early when things go "off track".

$\endgroup$
7
  • $\begingroup$ Based on that way you have characteriZed the problem, it should be time to even. Once you arrive at that time you create a 0/1 flag to alert the decision maker. $\endgroup$
    – forecaster
    Oct 30, 2022 at 18:30
  • $\begingroup$ I have a solution for you, give me a day or so to respond. Its little bit involved, I would use empirical bayes to respond since you mentioned Efron. $\endgroup$
    – forecaster
    Oct 30, 2022 at 20:23
  • $\begingroup$ thanks; looking forward to proposal. @forecaster FYI I updated OP a bit based on feedback added clarification on what we try to avoid (using domain specific knowledge based on features to alert). $\endgroup$ Nov 1, 2022 at 0:17
  • $\begingroup$ It sounds like you want to know the distribution of wait times for a patient with a certain set of covariates and then ring an alarm when you get to a certain level of unusual-ness. Is this accurate? $\endgroup$
    – Dave
    Nov 2, 2022 at 1:31
  • $\begingroup$ Yes, that's a good tl;dr. The key here is that estimating a Pr(wait time | covariates) for every patient is not the problem (that's easy; just train your favorite survival model / ML model). My question is that given you have that distribution how do you 'ring an alarm' about being overdue, when all predictive distributions [even the oracle ground truth distribution] will tell you at any given moment that Pr(time until recovery | features at time t) is due (far) in the future. As @seanv507 points out below it might boil down to including a latent state variable 'alert' of some sort. $\endgroup$ Nov 2, 2022 at 1:41

1 Answer 1

0
$\begingroup$

I think you should try to come up with the simplest problem first. It's not very clear what you are trying to do.

I think what you are having problems with is you have an undefined latent dependent variable. Eg more than 50% chance of failure in the next month/year etc.overdue means that you have reached this threshold in the past(but no-one was monitoring it)

So in this example you would have 2 survival models. One to predict probability of failure, one to predict reaching probability threshold given haven't reached it yet.

In practice I would imagine doctors define an explicit metric. EG your cholesterol is above X is a proxy for you reaching a certain probability of having a heart attack in the next year.

$\endgroup$
1
  • $\begingroup$ thanks for comment; I updated OP to clean it up a bit and focus on medical example only. yes, agreed that thinking of this as a latent variable problem is a good way to approach it. Re doctors defining metric: that's exactly what I want to avoid as it does not yield best outcomes. They make sense as domain-specific rules, but there is def room for improvement given all the other measurements/information available about the patient. That's what my question is about -- how do we incorporate this information to avoid relying solely on gut feeling/rule of thumb $\endgroup$ Oct 31, 2022 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.