I'm working with a survival model to evaluate its accuracy on a new dataset that was not involved in training or initial evaluation. This dataset is a truly independent dataset. Within this dataset, most of the events occur within just under 2 years (>90%), but we have additional observations that occur beyond that 2 year timepoint. Additionally, the median time of follow up is just under 2 years.
At this stage, I have chosen to calculate a 2-year risk score from the model for this new dataset, but there have been questions regarding how to properly identify those observations for which an event happened soon after the 2 year mark. Below are the two main approaches that I've tried to present in an unbiased manner.
One argument is that events are events, and given their close proximity to the 2-year mark, they should be considered events for all further evaluations as their 2-year risk scores are likely higher than any non-events observations within the study.
The alternative argument is that those observations for which events occurred after the risk score time should be counted as non-events regardless of their proximity to the calculation date. In other words, if an event occurred at 2 years and one day, that's still a non-event at 2 years and should be considered as such.
A third alternative consists of having three groups: non-events, events within 2-years, and events after 2-years.
Another suggestion has been to calculate the risk score for the maximum follow up time for an event observation, in this case 2.4 years thus covering all possible event times. The argument against this is that some of the censored (and thus non-event observations) may have had events that we do not know about, potentially altering the non-event group's risk scores.
I would appreciate any suggestions regarding how to approach this problem.
Some extra details following the response by Todd:
The original model is an AFT model and the original data had a 5-year follow up time. I'm using this model now in a new dataset which has a shorter follow up time overall, but is similar in composition to the original dataset. This is where the 2-year value for risk prediction is coming from, and it is simply a parameter within the AFT model.
For validation we have taken various approaches including test of calibration including: Hossmer-Lemeshow test using quantiles of risk scores compared to actual event rate within those quantiles; and most relevant to this questions, a discrimination test (discrimination slope?) comparing the distribution of risk scores for those observations considered non-events to those considered events.