Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

This question is with regards to using a test data set to validate an imputed Cox model using R. With a non-imputed data set I would use val.surv() from rms, but I'm not sure how/if I can use it with my multiply imputed data set.

Further explanation: I created a predictive Cox PH model for 5-year RFS, and also used the mice package to multiply impute some missing data in the training data set. I then used fit.mult.impute() in Dr. Frank Harrell's excellent Hmisc package to obtain the pooled model. I have a data set that I would like to test the model on, but I am not sure how best to validate the pooled model.

Multiple imputation is a common procedure that many researchers utilize, so there must be a way that R users are validating their models with imputed data. I would like to know what functions/avenues are available for me to test this pooled model with my validation data set? Here is some sample code to work with:

library(rms)
library(survival)
library(mice)

remove(veteran)
data(veteran)
veteran$trt=factor(veteran$trt,levels=c(1,2))
veteran$prior=factor(veteran$prior,levels=c(0,10))

#Set random data to NA 
veteran[sample(137,4),1]=NA
veteran[sample(137,4),2]=NA
veteran[sample(137,4),7]=NA

impvet=mice(veteran)
survmod=with(veteran,Surv(time,status))

#make a CPH for each imputation
for(i in seq(5)){
    assign(paste("mod_",i,sep=""),cph(survmod~celltype+karno,
        data=complete(impvet,i),x=T,y=T))
}

#Now there is a CPH model for mod_1, mod_2, mod_3, mod_4, and mod_5.

pooled_mod=fit.mult.impute(survmod~celltype+karno,cph,impvet,data=veteran,surv=T)

#Here is a test data set.
remove(veteran)
test_dat=data.frame(trt=replicate(500,NA), celltype=replicate(500,NA), time=replicate(500,NA), status=replicate(500,NA), karno=replicate(500,NA), diagtime=replicate(500,NA), age=replicate(500,NA), prior=replicate(500,NA))
for(i in seq(8)){
test_dat[,i]=sample(veteran[,i],500,replace=T)
}

#Now there is a pooled model, "pooled_mod", and a test data set, "test_dat".

I'm looking forward to hearing about the R methods that can help in this situation.

share|improve this question
2  
You might be interested in the Amelia package which has some features for assessing imputed data built in. The vignette for the package is pretty comprehensive (pdf): cran.r-project.org/web/packages/Amelia/vignettes/amelia.pdf –  smillig Jan 10 '13 at 9:33
add comment

1 Answer 1

up vote 3 down vote accepted

I would suggest that you repeat every step that you would otherwise do on the non-imputed data for the first imputed data set. For example:

# fit the model on the multiply imputed data
fit <- with(impvet, cph(survmod ~ celltype + karno, x = T, y = T))

# take out the first imputed data set
data <- complete(fit, 1)

# take out the first fitted cph model
mod <- fit$analyses[[1]]

# use your val.surv() steps here on the first imputed data set
...

Of course, you can also study the second, third, … dataset.

share|improve this answer
    
I'm not sure I understand what you mean. Are you saying that I should validate only the model that is based on one of the imputations? Surely there is a way that R users are validating their models pooled from multiple imputations? Or is this a circumstance where SAS trumps R? –  JJM Jan 10 '13 at 18:59
    
JJM, I understand your confusion. Preferably you should do all m analyses, and pool the results. My interpretation of your question was how to apply val.surv() in the context of multiple imputation. All I'm saying is: Apply it to the m imputed data sets, and not to "the pooled model". –  Stef van Buuren Jan 10 '13 at 19:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.