0
$\begingroup$

I've built very-well performing survival model (weibull proportional hazards model for interval censored data, modelled with IcenReg) with many covariates, some interaction terms and some splines. The model's coefficients are very hard to interpret, if not impossible- there's a lot of collinearity between the variables too. I would like to show how the model makes decisions- e.g. how to split the data to get good separation of predicted risk?

My initial thoughts were decision trees- and they work to an extent. I get 4 sensible levels out of the tree if i play around with rpart::rpart.control, but it only splits according to three variables... What can I do to show how the model makes its predictions?

$\endgroup$

1 Answer 1

1
$\begingroup$

"Feature importance" isn't as well defined as you might hope. There are ways to evaluate the contributions of predictors to a model fit by maximum likelihood, like your icenReg Weibull model, but they typically don't extend well to new data.

For a predictor included in multiple coefficients, like a continuous spline fit, a multi-level categorical predictor, or a predictor involved in interactions, you can do a joint Wald test on all the coefficients involving it. That's sometimes called a "chunk test." That test uses the coefficient estimates and (the inverse of) the corresponding part of the coefficient variance-covariance matrix to evaluate whether any of the coefficient values is significantly different from 0. This answer provides some detail that you should be able to adapt to the coefficients (coef(fit)) and the variance-covariance matrix (vcov(fit) of your model.

The resulting test statistic under the null hypothesis is distributed as chi-square with a number of degrees of freedom equal to the number of coefficients. As the expected value of the statistic under the null equals the number of degrees of freedom, the chi-square statistic for the "chunk test" on the predictor, minus the number of degrees of freedom, is a reasonable choice for an overall estimate of a predictor's contribution to the model.

As Frank Harrell discusses in Section 5.4 of Regression Modeling Strategies, however, the rankings of predictors aren't typically stable over multiple data samples. Even models of repeated bootstrap samples from the same data set can provide wildly different "feature importance" rankings. Try that on your own data and modeling process.

If some of your variables are highly collinear, you might consider re-formulating your model to combine sets of highly related variables into single predictors for the model first. If you do that without considering the associations of those variables with outcome, you don't affect the interpretation of p-values and the like from the resulting model, while you cut down on the number of degrees of freedom used up in the modeling.

If there's a reasonable biological/clinical argument for combining multiple variables this way, that approach also simplifies model interpretation. Think, for example, about the "Tumor, Node, Metastasis" combinations used to evaluate overall disease Stage in cancer. Harrell discusses the principles of this "data reduction" in Section 4.7 of Regression Modeling Strategies and illustrates application later in the document.

The above assumes that you have a validated model to start with. With "many covariates, some interaction terms and some splines," there's a risk of overfitting the data unless you have a correspondingly large number of events in your data. It's the number of events, not the number of total cases, that provides power to a survival model.

As I recall, your data set has a large number of cases but with only a small percentage having the event. You typically need on the order of at least 15 events per coefficient that you are estimating to avoid overfitting. If your model is overfit, then any "feature importance" culled from it won't extend well to new data at all, even beyond the random-sampling and collinearity issues discussed above.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.