I am doing survival analysis. There is a dataset of items
(id, group_id, observed lifetime, censorship status), each item belongs to a certain group. Each item is unique, but this uniqueness is unobservable, unpredictable and uncontollable, so we cannot base our predictions on it. Let's call that dataset
items dataset. Each group has features easily obtainable as a separate dataset of groups
(group_id, group_features). Let's call that dataset
One of the main goals of the research is to be able to predict expected lifetime of items of a certain group from its features and to be able to explain why (I utilise SHAP for that). In fact order predictions make more sense, but it is the next stage of the research. At the current stage I just want pretty accurate lifetime expectations.
I have combined the data from
items dataset by
gid and got a set of subdatasets of failure times belonging to each group. I have fitted a Weibull regression against each of those subdatasets of and got ρ and λ modelling survival of each group. I call this operation an
aggregation. It maps
items dataset to the dataset
(gid, ρ, λ, #items with that gid in items dataset). Then I add there the data from
Then I try to fit a model predicting ρ and λ with XGBoost using the objective
reg:squarederror and count of items in each group as a weight.
Then I predict the expected lifetime of an item of a group using the formula
XGB<ρ>(...) means calling the trained XGBoost model for the column
ρ passing it the group's feature vector.
- CV error (only XGBoost one) is tremendous even after 10000 iterations of hyperparameter optimization
the composite model predicts values which are completely mad. For example typical non-censored lifetime of an item is about 1500 days, but the composite model predicts 2 days as an expectation.
Am I doing the right thing? Is it possible at all to predict regression coefficients and then use them to predict expectation? Should I use count of items in groups as weights?
Should I modify XGBoost objective plugging there hessian and gradient of expectation of Weibull distribution and try to predict the expectation directly by feeding it directly with
items datasetaugmented with the data from
groups datasetand having a gradient for each item individually, rather than just predicting Weibull distribution coefficients and then expectation from them? Can I expect them to have the comparable performance?