4
$\begingroup$

I am trying to interpret the results of a Cox regression; I am doing a PhD in medicine. I love statistics but my question is still pretty basic, I think, and I did not find an answer in previous threads.

I have to compare different models (just a couple of predictors in each; predictors are different, but sometimes the same predictor appears in different models; say, A+B, A+C, A+D, B+C) toward the same time-to-event variable.

How do I choose the “best” model? I am studying the underlying statistical principles, but I still don’t get whether I have to look at

  • Which model has the highest log likelihood;
  • Which model has the best p-value of the likelihood ratio test (LR chi2);
  • Which model has all p-values of the HR coefficients (beta’s) of the covariates significant;
  • Or any combination of the above (for instance, only consider the models where both the LR and the beta-coefficients of all covariates are significant, and among them choose the one with the highest log likelihood).

Or is there another statistical technique to do it? I am using Stata 11.0.

I understand that the significance of the LR and the significance of the beta-coefficients test different things, but still I need to select the model with, say, the “best predictive ability” or the “strongest association”.

Thank you in advance for your help! Luca

$\endgroup$
  • $\begingroup$ Working on an answer, but a clarifying question: Are you actually looking for the best prediction, or something like an unbiased estimate of association controlling for confounding variables? These two goals are...sometimes at odds with each other. $\endgroup$ – Fomite Sep 30 '13 at 16:05
  • $\begingroup$ Thank you! I am just looking for the best prediction. But if you have remarks on how to do an estimate of association with control for confounding it might also be appropriate and interesting to hear. $\endgroup$ – torwart Sep 30 '13 at 16:10
  • $\begingroup$ Sadly then, I think I'll let someone else take this one - prediction isn't something I do much of, and wouldn't want to lead you astray. $\endgroup$ – Fomite Sep 30 '13 at 16:18
  • $\begingroup$ I would still like to hear you about estimating association, though. Even just some concise remarks! Thanks! $\endgroup$ – torwart Sep 30 '13 at 16:26
7
$\begingroup$

Disclaimer: As in the comments, these are not ways to ensure best prediction, but rather the musings of an epidemiologist on model building for survival models trying to elucidate the relationship between an outcome O and an exposure E with a number of covariates:

The goal of these is not actually to make the best predictive model, or the strongest association, but rather to come up with a model that contains all the variables necessary to have an unbiased estimate of the effect of E on O (given no residual confounding - i.e. we didn't forget/overlook/have no idea a particular thing is important), without including anything else.

Because your models aren't "nested", i.e. you're not comparing "A, B and C" versus "A and B" versus "Just A", you really can't use a direct comparison of the log likelihoods, including the likelihood ratio test.

Making your modeling decisions based on p-values is also fairly perilous - a huge amount of discussion could be made about this, but I'd suggest as starting literature checking out a copy of Modern Epidemiology 3rd Edition, or browsing some of the works of Sander Greenland, or perhaps Charles Poole. That should put you off p-value model selection fairly quickly :)

If you're just looking for the "best fitting" version of non-nested Cox models, you could use something like Akaike information criterion (AIC) which Stata should report, or the Bayesian information criterion (BIC). These give you a decent picture of the relative strength of each model fit - you're looking for the model with the lowest AIC or BIC. These give you a picture of the predictive power of the model compared to the amount of variables in it, trying to give a balance between model parsimony and fit. I tend to use this if what I'm trying to decide on is the form of a variable to include (i.e. should I use A, or also include a term for A^2?). But not so much in the "Which variables do I include" stage.

The way I decide on variables is using a mix of these:

  • I build what I believe is a working causal model of the relationship using a Directed Acyclic Graph (DAG) to show all the relationships between E, O and my variables of interest. There are many introductions on how to do this, and some would argue that once you have done a DAG and found the variables you need to control for (see said online tutorials) you're done. My confidence in this varies based on whether or not I'm working in a known, well-studied area, or am paving new territory.
  • If I don't want to be finished there, or I'm not sure about some of my choices, I might use a change-in-estimate approach, including variables that change my estimate of the association between E and O by more than 10% or something of that nature. This lets you keep the variables that have an impact on your estimate, but get rid of those that don't, even if they might theoretically have been important because of your DAG.
  • Finally, sometimes I do just use a p-value cutoff, but I tend to make this extremely generous - I'm not looking to only include variables with small p-values, but any variable that even faintly shows that it might have importance, so my cutoff is something like p < 0.25.

I'd again recommend a copy of Modern Epidemiology 3rd Edition, they've got a very good treatment of model selection.

$\endgroup$
  • $\begingroup$ Thank you, these are very interesting considerations. The DAG is really, really intriguing. I was aware that comparing p-values has little meaning; it is one of the considerations that led to my question. And I was actually studying Modern Epidemiology before I switched to "Epidemiologic methods" of Koepsell/Weiss. But now you recommend it so warmly I will give Rothman another chance. $\endgroup$ – torwart Oct 1 '13 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.