Cox Regression - Output at Individual (observation) Level I am building an attrition model. I was able to run the following cox regression. The rate of attrition looks very low, but I am only basing that on the plot I ran below. I would like to get a prediction for each case if they will attrit* and when**. Is that possible with the coxph function?
cox3 <- coxph(Surv(Raw_months, Attrit) ~ Raw_age + Ethnicity_dummy + Interim_grade + Retirement_dummy, data = Copy_of_fake_data_V2)
summary(cox3)
cox_fit3 <- survfit(cox3)
plot(cox_fit3,main = "cph model", xlab="months")

In the table below, * and ** are the new columns I want for each person in the data set




Unique Id
Raw_months
Attrit
var. 1
var. 2
Will they attrit?*
When(months)**




1
324
0
0
6
1
5


2
14
0
1
4
0
67


3
62
1
1
3
N/A
N/A



 A: This follows up on a related question that states an interest in predicting attrition over the next 5 years. The Cox model in this question is for time to attrition, starting from the date of hire into a firm.
This starts with an answer to the question as posed, even though that might not be the best way to approach the overall attrition-prediction problem. At the end will be a brief discussion of a better approach.
Outline of solution
First, you need to get a survival-time estimate for each individual based on covariate values and your model. With your model, the survival-time estimate will be relative to the date of hire. Functions with names like predict() or quantile() can make such estimates. We'll discuss below what to use as a survival-time estimate.
Second, to predict attrition over the next 5 years, you need to convert the survival-time estimates to put them in a scale of months from a new fixed reference time shared by all. For simplicity, use the date of the last data pull mentioned in the related question, 11/2020, as that reference. For example, if the survival-time estimate for an individual is 180 months from date of hire and that individual had 140 months of service as of 11/2020, then the estimated survival beyond the new reference time would be 40 months. That can give you estimated survival beyond the new reference for all individuals, filling in your When(months) column.
Then, if you are interested in 5-year survival beyond 11/2020, your prediction for the Will they attrit? column would be whether the predicted survival beyond that date is less than or greater than 60 months.
Handling predictor variables
Your model implicitly is based on predictor values that were known at date of hire. For example, Ethnicity_dummy and Raw_age (which I take to represent age at hire) fit that requirement. The Interim_grade and Retirement_dummy, in contrast, seem to be predictors that aren't fixed at the date of hire and that could change over time.
If you nevertheless include the current values of Interim_grade and Retirement_dummy in your model, (1) you are effectively assuming that those values held throughout the time since hiring and (2) you are running a big risk of survivorship bias. If you want to include such predictors you need to set up and analyze the data to accommodate time-varying covariates. Your outcome value would then be Surv(startTime, stopTime, event), potentially with multiple rows per individual, where startTime and stopTime represent the period over which a certain set of predictor values hold for an individual. Note that this might complicate your predictions for the next 5 years, as you will then need to estimate future changes in those covariate values.
Consider adding other predictors to your model and more flexible modeling. The year of hire, for example, could be included to see whether there were changes over calendar time in the attrition tendency. For that, Raw_age, and other continuous predictors you should consider something beyond a simple linear predictor form; restricted cubic splines, as implemented for example in the R rms package, allow the data themselves to help determine the proper relationship between the predictor and outcome. You also could incorporate interaction terms between predictors, to allow for the influence of one predictor on outcome to depend on the value of another predictor. With about 2000 events you might be able to include up to 100 predictors (including their interactions) in your model without overfitting, so think big in terms of what to include.
Model-type and survival-time choice
The considerations above hold regardless of the type of survival model that you choose. Your work will probably be easier if you use a parametric survival model (e.g., survreg() in the survival package or psm() in the rms package) instead of a Cox model.
With a properly fit parametric model, things like estimated mean survival for a set of predictor values can be determined analytically. You also aren't constrained by the proportional-hazards assumption of a Cox model, as some parametric forms like the log-normal don't require that, modeling accelerated failure times instead.
A Cox model is best for evaluating relative hazards, as the baseline hazard is purely empirical. If the observations at the longest survival times are censored (which seems to be the case with your data), you can't even get a true mean survival time estimate. (You can get a "restricted mean," calculated out to a specified time before the last event.) Provided that your data go out far enough in time (as yours seem to) you can nevertheless get estimates of predicted median survival from a Cox model, so you could continue with the Cox model if its assumptions are met and a median estimated survival or a restricted mean survival are suitable for your predictions. I'm not sure whether median or mean survival-time estimates would be best from your perspective of 5-year predictions going forward, but with 2000+ events you should be able to play around with the data from the prior 5 years to see what works.
A better approach
The above attempt to provide a specific survival time for each individual leads both to a false sense of precision and to some predictions that won't make sense. All individuals with the same set of covariate values will have the same predicted survival after hire, even though there is a wide distribution (over decades, in this case) of survival times for any given set of covariate values. Furthermore, you will find a lot of individuals whose predicted survival time will be earlier than their survival to date. So their predicted times-to-attrition will be negative!
What you really want to know is the probability that an individual will survive for 5 more years, given survival up to the present.  Pooling that information over all individuals of interest, you will get the overall survival (or attrition) for the next 5 years. The considerations above for building the model still hold, but your application of the model is more likely to be useful if you use this approach.
The survival model provides the probability of survival as a function of time from the date of hire. You should take advantage of those probability estimates. Instead of estimating a particular survival time for each individual, use the survival model to estimate his or her probability of having survived from hire through the time of the data pull at 11/2020. Then estimate the probability of survival for that individual from hire through 11/2025. The ratio of the second probability to the first is the probability of surviving 5 more years, given survival through 11/2020. The 5-year attrition probability is 1 minus that value.
