I am wondering if it is appropriate to use the term "multiple comparison problem" when applied to multiple imputation. I know that the multiple comparison problem arises when we have one set of data and ask many questions about it. Is this theoretically the same thing as having multiple data sets, and asking the same question on each dataset?

The reason I ask is because I have a MI dataset, and want to run a log rank test on each of the 50 datasets, but I don't believe that running the test on each dataset and then pooling is valid (because of the multiple comparison problem).

  • $\begingroup$ What exactly do you impute? The group variable or something else? $\endgroup$ – Theodor Nov 2 '15 at 12:06
  • $\begingroup$ Correct. In my original dataset, the survival time and censoring indicator are fully observed, but the treatment/group needs imputation. As well, in another instance, I have that same setup, but a variable is imputed that I subset on (ie coxph(Surv(time,cens)~treat,subset=(receptor==TRUE)) $\endgroup$ – RayVelcoro Nov 2 '15 at 14:20

I found this which might be of interest.

On another hand, I would find it more meaningful to pool the estimates of the log-hazard ratio and obtain their pooled standard error according to Rubin's rules (available, for example, here), and then conduct a t-test for the pooled estimate.

For other practical purposes, I'd also report the range of the p-values obtained form the multiple imputation, if pooling the estimate proves to be too complicated.

  • $\begingroup$ I am currently doing the Wald like test as you suggested in the second paragraph. My logic is that we want the pooled log rank test (from kaplan meier curves), but since this is not normally distributed, we can run a cox model (equivalent with no ties), and then to a Wald test, since the score test is the log rank test, and the wald test is asymptotically equivalent to it. I'm just afraid that. Besides, pooling on $\chi^2$ doesn't seem to work well. I'm just afraid of the theoretical soundness of this method. $\endgroup$ – RayVelcoro Nov 2 '15 at 16:38
  • $\begingroup$ Why wouldn't you run Cox models, estimate a regression coefficient (log-hazard ratio) from every imputed data set. The estimate of this, because it is a maximum likelihood estimate, has an asymptotic normal distribution. Then you pool the estimates, obtain the pooled standard error, and use that? $\endgroup$ – Theodor Nov 2 '15 at 16:44
  • $\begingroup$ Sorry, I don't think I explained it well. What I did was run a cox model on the treatment only (as a proxy for the kaplan meier curve, in order to get the log rank test), then pooled. But since we can't get a score (logrank) or LR test on this pooled data, I resorted to using the pooled estimate to do a Wald test. So I did do what you said (assuming in the last sentence, you mean "use that" to mean run a wald test). I'm just wondering what others will say about this...because after all, we do have access to the log rank test (but only on the individual dataset level). $\endgroup$ – RayVelcoro Nov 2 '15 at 16:52
  • $\begingroup$ I understand now. The log-rank test is essentially the score test for a binary variable. I'll have to think more about this. Let us know if you figure something out! $\endgroup$ – Theodor Nov 3 '15 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.