# Prediction of regression coefficients with XGBoost

I am doing survival analysis. There is a dataset of items (id, group_id, observed lifetime, censorship status), each item belongs to a certain group. Each item is unique, but this uniqueness is unobservable, unpredictable and uncontollable, so we cannot base our predictions on it. Let's call that dataset items dataset. Each group has features easily obtainable as a separate dataset of groups (group_id, group_features). Let's call that dataset groups dataset.

One of the main goals of the research is to be able to predict expected lifetime of items of a certain group from its features and to be able to explain why (I utilise SHAP for that). In fact order predictions make more sense, but it is the next stage of the research. At the current stage I just want pretty accurate lifetime expectations.

I have combined the data from items dataset by gid and got a set of subdatasets of failure times belonging to each group. I have fitted a Weibull regression against each of those subdatasets of and got ρ and λ modelling survival of each group. I call this operation an aggregation. It maps items dataset to the dataset (gid, ρ, λ, #items with that gid in items dataset). Then I add there the data from groups dataset.

Then I try to fit a model predicting ρ and λ with XGBoost using the objective reg:squarederror and count of items in each group as a weight.

Then I predict the expected lifetime of an item of a group using the formula

$$E(Weibull(ρ=XGB<ρ>(...), λ=XGB<λ>(...)))$$,

where XGB<ρ>(...) means calling the trained XGBoost model for the column ρ passing it the group's feature vector.

Unfortunately,

• CV error (only XGBoost one) is tremendous even after 10000 iterations of hyperparameter optimization
• the composite model predicts values which are completely mad. For example typical non-censored lifetime of an item is about 1500 days, but the composite model predicts 2 days as an expectation.

1. Am I doing the right thing? Is it possible at all to predict regression coefficients and then use them to predict expectation? Should I use count of items in groups as weights?

2. Should I modify XGBoost objective plugging there hessian and gradient of expectation of Weibull distribution and try to predict the expectation directly by feeding it directly with items dataset augmented with the data from groups dataset and having a gradient for each item individually, rather than just predicting Weibull distribution coefficients and then expectation from them? Can I expect them to have the comparable performance?

• If your feature set is identical, what is the point of using XGBoost? You essentially just estimate a baseline hazard. If you don't trust the Weibull assumption use a non- or semi-parametric model for the baseline hazard. XGBoost is only advantages for complex feature relationships with the outcome May 31 '19 at 13:44
• Thank you for commenting. The identical items are items of the same group. There is a dataset, it contains items of different groups. I fit Weibull regression for each group, get the coefficients. I call it aggregation, it transforms 2 datasets : one of items (id, gid, lifetime, censorship) and one of groups (gid, group features) into the dataset of groups (gid, group features, ρ, λ). Features non-specific to a group but specific to an item are unobservable. The goal is to predict expected lifetime for the groups not in the dataset at all. Against the resulting dataset the model is fitted. Jun 1 '19 at 12:27
• It seems like you are throwing away a lot of information by lumping together cases into groups and then doing group-wise weighted fitting. An earlier version of this question indicated that within each group having common feature vectors there were some censored and some un-censored cases, and that there were different failure times among the un-censored cases within each group. Those differences within groups are important for survival modeling. Details about numbers of cases, events, groups, and why you expect squared error to be useful to minimize in survival analysis would help.
– EdM
Jun 1 '19 at 15:47
• 1. Doesn't Weibull regression deal to censorship and intragroup difference itself? Isn't the data about censorship and individual items lifetimes redundant for XGBoost model since all the instances of the same group have the same feature vector? Jun 1 '19 at 20:46
• I'm still unclear on what you are actually trying to achieve. Could you edit your question stating clearly what the objective of your analysis is (what are you trying to find out) and what are the features/targets, ideally without reference to Weibull regression and XGBoost first. Then explain how you tried to solve your objective using Weibull model and XGBoost. Jun 2 '19 at 11:03

As I understand the question, you have information on survival of individuals as a function of group membership, but don't have detailed information about the features associated with each individual. You do, however, have information about covariates associated with each of the groups.

First, you've done separate Weibull models on each group, which (as @adibender noted in comments) is simply modeling the baseline hazard for each group. Although Weibull modeling can often be useful, it's not clear whether you have even documented that Weibull models are appropriate for your data. For example, Weibull models don't allow for non-monotonic hazards as a function of time, so one potential explanation for your difficulty is that Weibull modeling is not appropriate for your data.

Second, parametric Weibull modeling of survival data, even when appropriate, is typically done with all cases at once. The model would identify a value of $$\lambda$$ common to all groups along with the differences among groups with respect to ρ. I can't prove that your approach (separate Weibull models for each group and then modeling the Weibull coefficients as a function of covariate value associated with groups) is necessarily wrong, but it doesn't seem to comport with the way that survival analysis in typically done.

If your interest is in the relations of the covariates to outcome, I see two ways to proceed.

One way would be modeling survival as a function of group first, then modeling group membership as a function of covariates. If all the information you have about survival is directly related to group membership, then the most direct survival model would be based on group membership. You then identify covariates most associated with the longest/shortest-survival groups.

The second way to proceed would be to accept the limitations of your data and simply assign to each individual the covariate values associated with that individual's group. That would remove the group membership from the analysis and allow direct (although clearly imprecise) modeling of how the covariates are related to outcome. If your group dataset has information on the probabilities of features associated with each group, you could treat this as a type of missing-data problem, using multiple imputation to obtain several sets of imputed data and thus incorporate the uncertainty in the individual-covariate associations.

With either of those approaches you would not have to use a parametric model, Weibull or otherwise. You would be able to test the appropriateness of the models (e.g, test for proportional hazards) in a way that your present approach doesn't seem to allow.

• Thanks fot the answer. In my case groups are models of equipment so memberships in groups are rigid and known in advance. Covariates are technical specifications of a model. Item in items dataset is a device which lifetime is known. The matters is complicated by the fact that not all possible groups are in the dataset and I try to predict expectation of survival for groups not present in dataset. If I understand right, you propose to just merge the datasets, making groups features items features and fit a model against that. I have already tried that without much success, Jun 6 '19 at 18:06
• , with big memory overhead and very bad performance and even with some numerical problems causing XGBoost to predict NaNs if the dataset exceeds some count of items. Though comparing to Weibull, Cox non-PH (with XGBoost predicting partial hazards instead of linear regression) worked pretty well (0.7 concordance, though the predictions were complete junk, it had predicted high lifetime expectation for some models known as faulty). Jun 6 '19 at 18:13
• I have tried to optimize that too by using harmonic mean, but that "optimized" model resulted in concordance 0.5. About imputation, I don't quite understand how it can help me here. I used imputation for missing data when a model (like stock lifelines Cox PH) disallows missing data (XGBoost was used for imputation), but witnout much success, XGBoost-powered Cox is the best model I have managed to achive on this task, and unfortunately that model is junk. Jun 6 '19 at 18:20
• @KOLANICH you could consider an extension of my first suggestion to cover groups not present in the data set. Model survival by group for present groups (combining all in a single analysis), and develop a multinomial model for predicting group membership from covariate values. Covariate values for new equipment models would then be mapped to group-membership probabilities. Those probabilities could allow some weighted-average of group survival predictions. The failure of my second suggestion, however, makes even that unlikely to work well; you might need to model interactions among covariates.
– EdM
Jun 6 '19 at 18:21
• Doesn't XGBoost model interactions? Jun 6 '19 at 18:23