0
$\begingroup$

I'm looking to compare the capacity of two models for predicting who does, and who does not, develop a certain infection during a hospital stay. Both models are based on the exact same patients, but the data is formatted differently.

The first model is a logistic regression model, based on a hospitalization-level dataset (1 record per hospitalization) where the outcome, y=1 indicates that a patient developed an infection before discharge and y=0 indicates a patient did not develop an infection before discharged. I have several different risk factors on the RHS of the model, including time at risk (i.e. the number of days they were hospitalized before they were discharged or acquired the infection), the age of the patient, and several others.

For the second model, I exploded the dataset so that one record represents a patient-day. The second model is a poisson regression model with follow-up time (# days since admission) as a categorical variable, so it corresponds to a piecewise exponential survival model. This model has a similar set of variables, but some of them are time-varying due to the added flexibility of the patient-day dataset format.

I'd like to compare the capacity of these models to predict who does, and who does not develop infection. My initial thought is to measure the concordance statistic for the logistic model. For the poisson model I would sum up the incidence rates to get a cumulative predicted incidence per hospitalization. It seems like an intuitive way of going about doing this comparison, but I haven't found anyone who has done this before.

Any critiques, suggestions for alternate approaches, or relevant literature would be greatly appreciated.

Best Kevin

Improve this question
$\endgroup$
3
  • $\begingroup$ In general, in you want to determine which of two models is the best at predicting, you'd need new data, a test data set, that was not used for training your model. Otherwise, you might just choose the most flexible of your models, even though it might be overfitting. $\endgroup$
    – swmo
    Commented Mar 5, 2015 at 23:38
  • $\begingroup$ In this circumstance, there is no variable selection going on. All of the variables I mentioned are included in the model, a priori. So overfitting shouldn't be an issue. $\endgroup$
    – Kevin Brown
    Commented Mar 6, 2015 at 0:25
  • $\begingroup$ Try looking here: en.wikipedia.org/wiki/Overfitting, I'm not talking about variable selection. To elaborate further: even if new data/holding out data is not feasible, you need to do something else to evaluate the predictive capabilities of your models. $\endgroup$
    – swmo
    Commented Mar 6, 2015 at 8:06

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.