3
$\begingroup$

Let's assume we have a prediction algorithm (if it helps, imagine it's using some boosted tree method) that does daily predictions for whether some event will happen to a unit (e.g. a machine that might break down, a patient that might get a problematic medical event etc.) during the next week. We can only get training and test data for a low number of units (e.g. 400 + 100 or so) across a limited period of time (e.g. half a year).

How would one assess prediction performance (e.g. sensitivity/specificity/AuROC etc.) of some algorithm (e.g. some tree method) in this setting on test data? Presumably there is a potential issue in that the prediction intervals overlap and even non-overlapping intervals for the same unit are somewhat correlated (i.e. if I can predict well for a particular unit due to its specific characteristics, I may do well on all time intervals for that unit, but this does not mean the algorithm would generalize well).

Perhaps I have just not hit on the right search terms, but I have failed to find anything published on this topic (surely someone has written about this before?!). Any pointers to any literature?

My initial thought was that perhaps naively calculated (i.e. just treating this as independent observations and predictions) point estimates of sensitivity/specificity might be fine, but that any problem would be more with the uncertainty around these? If so, could one just bootstrap (drawing whole units with replacement) and get decent assessments that way?

Edit to illustrate what data may look like: Here is some made up illustrative data that illustrates what data I might have with overlapping 7-day time intervals (units 1, 2, 3,...,500, time windows [here show in terms of their start and end days to illustrate the overlap], the outcome of whether an event occured in that 7 day period [Yes/No] and lots of predictors, say, a few hundred). Some of the same problem with correlations within units would sitll be there, if intervals were non-overlapping e.g. day 1-8, 9-15, 16-22 etc. \begin{array} {|r|r|} \hline \text{Unit} & \text{start day} & \text{end day} & \text{Outcome} & \text{predictor 1} & \text{predictor 2} & \ldots \\ \hline 1 & 1 & 8 & \text{Yes} & 1.23 & 16 & \ldots \\ 1 & 2 & 9 & \text{Yes} & 1.5 & 16 & \ldots \\ 1 & 3 & 10 & \text{Yes} & 1.8 & 16 & \ldots \\ 1 & 4 & 11 & \text{No} & 0.5 & 16 & \ldots \\ \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\ 1 & 176 & 183 & \text{No} & 0.21 & 10 & \ldots \\ \hline 2 & 1 & 8 & \text{No} & 0.12 & 2 & \ldots \\ 2 & 2 & 9 & \text{No} & 0.05 & 2 & \ldots \\ \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\ \hline 500 & 1 & 8 & \text{No} & 0.17 & 11 & \ldots \\ 500 & 2 & 9 & \text{Yes} & 0.53 & 11 & \ldots \\ \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\ \end{array}

$\endgroup$
  • $\begingroup$ Are you using a single model to make predictions for all units? Have you thought of using unit ID as a feature? What about multi-level modeling? en.wikipedia.org/wiki/Multilevel_model $\endgroup$ – Bar Feb 14 '18 at 11:39
  • $\begingroup$ We plan on a single model in the sense of an ensemble of trees. You can also look at that as multiple models, but it's not separate models per unit ID. Does your comment on multi-level models refer to having separate models by unit ID that get regularized (i.e. only different to the extent support by data)? That might be an attractive idea if computationally feasible (even for completely new units we would know how much extra uncertainty this adds), while using unit ID as a predictor in the traditional sense may be less relevant (we aim to predict for completely new units without a history). $\endgroup$ – Björn Feb 14 '18 at 12:00
  • $\begingroup$ Maybe I'm misunderstanding you problem setup then. Could you provide an example data point in your question? It's important to understand which are the dependent and independent variables. $\endgroup$ – Bar Feb 14 '18 at 12:02
  • $\begingroup$ @Bar I've added some illustrative data to show what I imagine this will look like. $\endgroup$ – Björn Feb 14 '18 at 13:59
  • $\begingroup$ (Why) do you need to use these overlapping time windows? $\endgroup$ – Ruben van Bergen Feb 14 '18 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.