I want to fit a machine learning model to a dataset which is basically a survival analysis with competing risks with several failure types (e.g. mortality causes). However, I want optimal predictions of either survival or of the specific failure types in order to draw conclusions about which predictors are most important to distinguish potential types of failure.
I have tried several methods and couldn't figure out how to do it in a sound way. One problem is that the different failure types have to a great extent similar predictors. For example age is a strong predictor for all failure types. If I run a survival random forest for competing risk almost all patients get assigned highest risk for the mortality cause with the highest incidence (so almost no separation). Similar results come up with cause-specific hazard modeling with boosted Cox (xgboost). I also expect that the same will come up with fine gray transformation (because you have to apply these methods once for every failure type and they just don't discriminate between censoring and a specific different failure type).
I also tried DeepHit but just couldn't get it to work (in part probably because the event rate is actually low, <30% die during the full follow up).
The best approach for me so far was a simple multiclass classification with xgboost (one class for surviving and one class for each type of failure). The problem here is that >60% of patients are censored before the end of the follow up time. So I would say although it is working (based on CIF graphs for the different failure types) it's statistically not as sound as I would like. To relieve the problem of missclassification of surviving patients due to censoring I additionally applied weights to surviving patients based on how long the patient was at risk before being censored. This gave me actually the best results in terms of separation of the different causes in CIF graphs.
My question is what the best way would be do this kind of analysis. And as a second question, whether this last method could be used or whether it is problematic.
Edit: I think a multi-state model as suggested by @EdM could be a solution here, is there a machine learning (e.g. tree-based, such as boosting) implementation for a setting with many predictors (I have >200 predictors)? Without understanding all the inner workings, the mentioned text by Therneau, Crowson and Atkinson mentions in the section 3.1 (for rate models):
The same Cox model coefficients can also be obtained by fitting separate models for the PCM and death endpoint, censoring cases that fail due to the other cause. Hazards can be computed one at a time. When computing p(t), on the other hand, all the rates must be considered at once, it is necessary to use the joint fit cfit1 above. We create predicted curves for four hypothetical subjects below.
Where p(t) is the probability for each state (e.g. cause of death) at time t. So if I understand it correctly a normal cause-specific hazard is calculated and from there the p(t), however I don't thinks this is the optimal approach with my dataset (as discussed above). Rather I think optimizing p(t) directly based on the available outcomes should optimize for variables most important to distinguish the different outcomes.
In Section 2 they present a competing risk model based on the Aalen-Johansen method. Not sure how this would perform. Is there a machine learning implementation of this method available?