0
$\begingroup$

I am doing ROC-curve analysis on my patient cohort, and I am wondering if it is statistically ok to use "death" and "recurrence" as the binary marker, even though these parameters will be influenced by time? Or do I have to do time-dependent ROC-analysis? (Does anyone here have experience with SurvivalRoc?)

So OS and RFS and the factors influencing them I have investigated using the KM-method and cox regression. The only factor significantly influencing OS and RFS is the variable/test I want to investigate further using ROC-curves (high test-score indicates worse prognosis.) I want to find the optimal cut-off of the test, and I am wondering if I can use "recurrence" or "death" as the binary marker. I will not use the ROC-curves to look at survival, but as a tool to find the optimal cut-off value for what patients should receive further treatment.

$\endgroup$
2
$\begingroup$

If you wait long enough we all die, so using "death" as a binary marker doesn't really make much sense in survival analysis. One might make an argument that "recurrence" would be OK as a binary marker, but then you're throwing out information about time to recurrence.

The survivalROC package in R, as you note, will do what you want for a specified predict.time. To work with censored data there has to be some smoothing to ensure a monotonic ROC, so you have to specify the method and parameter values for the smoothing. I found it easiest to specify a span value, effectively the fraction of the data, ranked ordered by the predictor, over which smoothing is done. Read the paper to understand the details.

Recognize, however, that a ROC does not represent the most appropriate way to evaluate or use a survival model. This Cross Validated page provides a good entry into discussion of alternatives.

In response to paragraph added in revised question

Your proposed use of an ROC curve to choose treatments of patients based on results of a single test is problematic in several ways.

First, if "The only factor significantly influencing OS and RFS is the variable/test I want to investigate," then you have a data sample whose results might be difficult to generalize to a different set of patients. Studies typically find relations of multiple clinical variables to overall survival (OS) or tumor recurrence-free survival (RFS). Such variables include age, smoking history, tumor grade, and cancer staging. If you found that none of these was "significantly influencing OS and RFS," then you seem to have had a very small data set or one that was otherwise restricted, for example a relatively homogeneous sample with respect to these clinically important variables.

Second, your focus on a single "significant" variable is extremely poor practice for prognostication. For prognostication you should not remove a variable from consideration just because it failed a test of statistical significance in a particular data set. This issue has been discussed extensively on this site, for example here.

Third, choosing an "optimal cut-off value" is fraught with danger, even if you move from your single-test result to a more realistic predictor combining multiple prognostic variables. Presumably there is some cost or danger to the proposed treatment for those at high risk, or it would be offered to all regardless of risk. What is the cost-benefit tradeoff, both from the clinician's and from the patient's perspective? As @FrankHarrell has put it, "ROC curves are not informative in 99% of the cases I've seen over the past few years." What you need is a way to estimate risk for a patient, in a way that can be combined with other clinical, personal, and practical considerations to inform a choice of treatment. A simple yes/no classification based on an "optimal cut-off value" does not accomplish this.

Fourth, your wish to use "recurrence" or "death" as binary variables without respect to time only makes sense in limited circumstances, and throws out all of the useful information about time to recurrence or death. Maybe you could consider this if you had essentially complete follow-up data on all patients up through a time by which, based on prior knowledge about this disease, almost all recurrences or disease-related deaths are likely to occur. But that's not often the case in cancer. Even in that case, why not use the tools provided by survival analysis to learn from all the information that you have?

You will be much better off moving away from your ROC curves toward the more complete survival modeling provided by tools like those in the rms package in R. These tools can be used to calibrate and validate your survival model and to produce nomograms that combine multiple variables for prognostication. Links from this answer provide related resources.

$\endgroup$
  • $\begingroup$ Thank you! I want to do ROC-curve analysis in addition to the Kaplan-Meier method and cox regression (determining OS and DFS, and the factors influencing them), to find the optimal cut-off for a specific test used to see what patients should be treated. I want to see the test's ability to predict recurrence in particular (AUC 0.82 when just plotting recurrence and not taking time into account). Is it ok to do that in this setting? $\endgroup$ – user92232 Oct 16 '15 at 5:09
  • $\begingroup$ It's risky to throw away information, and what you propose seems to throw away information both about time and about clinical variables other than your specific test. There are much better ways to accomplish what you want. Please edit your question to include the information from your comment (comments can be lost, unlike questions), and also say how many recurrences and deaths are in your data set, and how many other clinical variables are typically considered in this situation. I'll revise this answer accordingly. In the meantime read about alternatives in the CV page I linked in the answer. $\endgroup$ – EdM Oct 16 '15 at 7:55
  • $\begingroup$ I have installed R with SurvivalROC, and done the analysis, but I am having trouble finding what span value to use (trouble finding an answer in the literature too), and I therefore wonder if you could explain the following sentence from you previous answer: "I found it easiest to specify a span value, effectively the fraction of the data, ranked ordered by the predictor, over which smoothing is done."? $\endgroup$ – user92232 Oct 25 '15 at 20:10
  • $\begingroup$ The default "NNE" method requires a choice of span or window. If you can use 'method= "KM"' (Kaplan-Meier) in the call to the function, you don't need to specify a span. Read the paper linked in the second paragraph of my answer to understand the difference between methods and how the NNE smoothing works. The example in the help page uses 'span = 0.25*nobs^(-0.20)', where nobs is the number of cases. You want to smooth just enough to avoid the problems introduced by censoring; evaluate in your own data set. And do consider the limitations of focusing too much on the ROC curve. $\endgroup$ – EdM Oct 25 '15 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.