Biostatistics - How to evaluate a prognostic predictive model at 5 different year points? So essentially I have patient data for a period of 5 years:
same patient each year
each person has 5 predicted risks (for each year) - e.g. in 2017 there risk was 2%, in 2018 the same patients risk was 8% etc
The equation predicts the risk of an event in 5 years
n is approx 100
All data inputs has been converted to numerical data
I want to asses how good the equation was at predicting the outcome. The outcome is already known.
Note: there are no false positives as the population being analysed all eventually developed the condition. This is due to the population, not because anyone was excluded. Would an AUCROC case be appropriate for this? Are there any other statistical tests available (other than sensitivity and specificity)
If I were to use AUCROC analysis, would it be appropriate to make 5 aucroc curves for each year?
edit: It is a prognostic predictive model that provides the risk of developing a condition in 5 years - its input is continuos variables. I seek to externally validate it
Thank you guys, Id appreciate any help
 A: The area under the receiver operating characteristic curve (AUROC) for each year is a good start at validation, but that doesn't provide information about the calibration of your model: how closely predicted and observed event probabilities agree. For that you need to set up a formal calibration curve.
In survival analysis like this (this seems to be a discrete-time survival model, but without censoring) you start by choosing a time for comparison. In your situation you presumably want to choose all 5 times of interest, which you would examine separately.
The simplest approach is to group cases by their predicted probabilities of having the event at the time of interest. Then plot the observed probability of events within each group against the predicted probability for the group. Ideally, you should get a straight line with a slope of 1 for the calibration curve thus produced. There is a way to do this without grouping, estimating the "observed" probability instead with a very flexible fit of event times as a function of covariate values and their interactions. See this page for an overview and links.
This approach can clearly be used for external validation.  You should be aware that it can be better to build a model with all of the data. Frank Harrell discusses the issues here. This type of approach can also be used for internal validation, for example by combining it with bootstrapping.
This page might be useful for further study; it's focused on Cox models, but many of the principles apply in general. A site search on "calibrat* survival model" will show several dozen additional pages related to these issues.
