5
$\begingroup$

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic injury (continuous), age (dichotomous, >= 55, <55). Probability of survival (Ps) are calculated for each patient:

$$ Ps = \frac{\exp\big(b_0 + b_1{\rm (physiology)} + b_2{\rm (anatomy)} + b_3{\rm (age)}\big)}{1+\exp\big(b_0 + b_1{\rm (physiology)} + b_2{\rm (anatomy)} + b_3{\rm (age)}\big)} $$

Used worldwide with the US derived regression coefficients, updated in 2005 and in 2009, it has had huge impact in trauma research. Foreign institutions has been able to benchmark their performance towards the US standard by use of the W-statistic (with 95% CI), expressing excess (less) survivors per 100 patients in own institution vs. the US standard. (W = (actual number of survivors – predicted number of survivors)/(number of patients/100). Some countries have used the same DV and IVs, however derived their own regression coefficients from their own population. Others have derived their own logistic regression model.

We have recently derived our own logistic regression model with the same DV, but different IVs. We have also implemented the TRISS model in our registry with the national US IV, but with IV coefficients derived from our trauma population.

Questions:

  1. I want to perform an external validation of the US model with national US IV regression coefficients. I want to compare the predictive ability/performance vs. our own model. How should we perform such a comparison? Compare ROC curves, deviance, or AIC? Is it a meaningful comparison?

  2. I want to perform an external validation of the US model, however with IV regression coefficients derived from our trauma population. I want to compare the predictive ability/performance vs. our own model. How should we perform such a comparison? Compare ROC curves, deviance, or AIC? Is this a more relevant comparison? How can we tell whether differences are significant or not?

  3. What about using Net Reclassification Improvement?

  4. I want to decide which model fits our data best. How?

  5. Can other institutions use our model for benchmarking?

$\endgroup$
2
$\begingroup$

It is important to note that the model you specified has no face validity, and it can easily be shown to be miscalibrated for patients of specific ages. The discontinuity the model specified does not occur in nature.

ROC curves do not have anything to do with model validation. High-resolution nonparametric calibration curves are all-important here. The R rms package val.prob function provides that plus many relevant statistics including the powerful Spiegelhalter test of calibration accuracy. Be sure to (in)validate the model's calibration in several age ranges.

$\endgroup$
  • $\begingroup$ If you're curious about the "high-resolution calibration curves". There is a discussion under "validation plot" in Steyerberg: "Assessing the performance of prediction models: a framework for some traditional and novel measures". He uses the same R package and results thus tend to look similar. $\endgroup$ – charles Mar 17 '14 at 18:16
2
$\begingroup$

I would read through some of Steyerberg's articles on the subject. Steyerberg: "Validation and updating of predictive logistic regression models: a study on sample size and shrinkage" may be of interest.

  1. External validation is important. But you can't fairly compare the US model to your model build with your data. That is an almost meaningless comparison since your model should outperform. (Though if it doesn't I guess that is meaningful)

    • some level of recalibration (as discussed in Steyerberg) article should be considered.
    • your model is overfit. some thought as how to address this should be considered.
  2. If you want to update/re-calibrate the US model be aware that you need a fairly large sample size to do this. This is why most just recalibrate the intercept+/- slope. Again this is addressed in above reference.

  3. To compare models usually you use some combination of the measures below. At a minimum calibration and discrimination should be evaluated. Utility is, unfortunately, rarely assessed.

    • global measure (scaled Brier)
    • discrimination (c-statistic)
    • calibration (often plot)
    • measure of utility based on domain knowledge or could consider Decision Curve (Vickers et al.)
  4. I don't think AIC or deviance are reasonable suggestions for model comparison.

  5. Regarding NRI, you should read Leening et al in Annals of Internal Medicine Jan 21, 2014 (+/- Vickers Comment on article in same) before considering using.

  6. Sure, other institutions can use your model. In much the same way, and with the same limitations and caveats, as they can use the US model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.