2
$\begingroup$

I'm working on a multicenter prospective cohort study investigating the predictive accuracy of a sepsis score (qSOFA) to predict 30-days in-hospital mortality and using Stata 17 ME. The score has four possible values (0–3) and is considered positive when it is ≥2.

I'm trying to build a well-mixed-effect model to assess the effect of qSOFA score ≥2 on the 30-days in-hospital mortality after adjustment of the component of the Charlson Comorbidities index (CCI). My concern is that elements of CCI are highly dependent on each other, violating one of the model assumptions (independency of covariates). For example, cancer is a function of an increase in age; diabetes is associated with an increased risk of chronic kidney disease, etc. Another concern is the number of parameters used in CCI is too many (may cause overfit), but I can combine some of them under one variable. For example, myocardial infarction, heart failure, and peripheral artery disease can be combined all under one parameter (cardiovascular disease 0/1)

My questions:

  1. Should I assess all possible inter-parameter interactions? And if so, should I put all these possible interactions in the model and then try to eliminate nonsignificant ones based on AIC, BIC, and wald test?
  2. Is it okay to combine relevant comorbidities under one variable to avoid overfitting?

Thanks Abdullah

$\endgroup$

1 Answer 1

1
$\begingroup$

My concern is that elements of CCI are highly dependent on each other, violating one of the model assumptions (independency of covariates).

There is no such assumption. Covariates do not need to be independent of each other. In the real world they seldom are. When you add an interaction term to a model, you are effectively adding a new covariate (the product of the interacting terms) that is not independent of the associated individual covariates.

This is related to "multicollinearity." This page notes the major problem with correlated covariates: the variances of coefficients are higher than they might be otherwise. Unless the model is misspecified or overfit, however, predictions from the model will be OK.

I'd recommend carefully reading Frank Harrell's course notes or book on "Regression Modeling Strategies." Chapter 4 of each goes into detail about how to deal with the situation that you describe. On that basis the answer to your question 2:

Is it okay to combine relevant comorbidities under one variable to avoid overfitting?

is certainly "Yes," if you do that in a way that doesn't use the outcomes for combining them. That's what Harrell discusses as "data reduction."

The answer to question 1

Should I assess all possible inter-parameter interactions? And if so, should I put all these possible interactions in the model and then try to eliminate nonsignificant ones based on AIC, BIC, and wald test?

is "No," however, without some important modifications. Harrell's general strategy is to: "decide the number of d.f. [degrees of freedom] that can be spent; decide where to spend them; spend them." The "number of d.f. that can be spent" (the number of unpenalized coefficients that you will fit) is a function of sample size; with a binary outcome, you could consider 1 d.f. for each 15 or so events in the minority class.

So you might do something like "assess as many inter-parameter interactions as you can that are expected to be outcome-related, consistent with the size of the data sample." The automated approach you suggest, throwing in all interactions and then pruning back based on apparent "significance," is itself a recipe for overfitting. You will fit the current data sample but the results are unlikely to extend to new data. See this page, for example.

The above discussion applies to unpenalized regression models; there are alternate approaches. A penalized modeling method like ridge regression can include a larger number of covariates, as their coefficients are reduced in magnitude in a way that minimizes overfitting. A tree-type model (random forest or gradient-boosted tree) can effectively identify and incorporate multiple outcome-associated interactions, if training is sufficiently "slow" to avoid overfitting. A downside with tree-based models is that it can be hard to interpret how individual covariates and their interactions are associated with outcome.

$\endgroup$
1
  • $\begingroup$ Thanks for your clear answer $\endgroup$
    – Abo7aneen
    Commented Dec 17, 2022 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.