13
$\begingroup$

I'm curious to learn more about choosing the confidence type within the survfit function in R. I know the options are conf.type=c("log","log-log","plain","none") but I'm not well versed in the process for choosing which option is the best.

Can someone describe the basic logic that goes into picking between the conf.type options? Or point me to a useful online resource? While I'm open to learn the math going on behind them, the most important thing right now is the factors that go into making the decision of which option to select.

I've run the function with each option and compared the confidence intervals and see that, for the most part, the are very close or the same but I'd like to understand more.

Thanks!

$\endgroup$

1 Answer 1

15
$\begingroup$

The typical formula for the Kaplan-Meier survival function is

$$\hat{S}(t) = \prod_{j|t_j \le t} \left( \dfrac{n_j - d_j}{n_j} \right)$$

where $t_j$ are the times at which failure $j=1,\dots,$ occurs, $n_j$ is the number at risk just before time $t_j$ and $d_j$ is the number of failures at time $t_j$.

Importantly, $0 \le \hat{S}(t) \le 1$.

Greenwood in 1926 gave us the formula for the standard error of $\hat{S}(t)$ through its variance:

$$\widehat{Var}\{\hat{S}(t)\} = \hat{S^2}(t) \sum_{j|t_j \le t} \dfrac{d_j}{n_j (n_j - d_j)}$$

Using the standard error derived from Greenwood's formula, we can calculate the $(\alpha \times 100)$ percent pointwise confidence interval of $\hat{S}(t)$ as

$$\left[\hat{S}(t) - z_{1-\alpha/2}\sqrt{\widehat{Var}\{\hat{S}(t)\}},\ \hat{S}(t) + z_{1-\alpha/2}\sqrt{\widehat{Var}\{\hat{S}(t)\}} \right]$$

This is what you get when you specify conf.type="plain".

There are several well-known problems with the Greenwood formula. One nasty one is that it may produce limits beyond the range of zero or one. That is to say, you can produce negative point estimates or point estimates exceeding 100%. In common practice, we just clip the confidence interval at zero or one and move on. Still the problem persists and, for teaching purposes, the Greenwood formula is relatively simple to calculate by hand and countless Master's students have learned to calculate confidence intervals for Kaplan-Meier curves on Greenwood's formula.

One way around this is to change the scale of the survival function $\hat{S}(t)$ by taking its log (that is, $\log\hat{S}(t)$. If we do that, it turns out that the confidence interval is similar

$$\left[\hat{S}(t) e^{- z_{1-\alpha/2}\sqrt{\widehat{Var}\{\hat{S}(t)\}}},\ \hat{S}(t)e^{z_{1-\alpha/2}\sqrt{\widehat{Var}\{\hat{S}(t)\}}} \right]$$

This is what you get when you specify conf.type="log" (the default). However, while we might have solved the problem of negative values for $\hat{S}(t)$, there is nothing that constrains the upper confidence limit from exceeding 1.

This brings us to the final attempt at taming the confidence limits. Here, we first take the negative of the log of $\hat{S}(t)$. Then, we take the log of that. That is, we try to calculate confidence limits for $\hat{S}(t)$ in the complimentary log-log scale: $\log \{-\log \hat{S}(t) \}$. This is what you get when you specify conf.type="log-log". The formula is rather unruly, so I won't replicate it here.

Of course, there are other transformations you can use to deal with the problem, but the log and log-log ones are the most in use.

Now, to your question. What confidence interval type should you use? There is no general consensus. The plain setting is great for its simplicity, the log setting produces variances that are stable, and the log-log produces intervals that are well-behaved, but may differ greatly compared to the other two settings. Finally, you must know that statistics packages differ in their default settings for the method of calculating confidence limits. R's default is the log setting while Stata and SAS use the complimentary log-log method.

I do hope this helps.

$\endgroup$
1
  • 1
    $\begingroup$ This is great! Very concise and easy to understand. Thanks! $\endgroup$
    – c.custer
    Commented Aug 9, 2018 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.