I selected for a univariate cox model an optimal cut-point based on the minimum p value of a log-rank test.
Are there any procedures to perform a sort of cross-validation or performance study for this cutpoint?
I'm using R so coding is not a problem
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Please say more about why you need a cut-point at all. Most advice on this site would be to keep continuous predictors as continuous (possibly with some transformation), as binning based on a cut-point throws away a lot of information and one tends not to find the same "optimal" cut point on multiple samples from the same population. $\endgroup$– EdMCommented Aug 30, 2016 at 13:05
-
$\begingroup$ Because it's an explorative study with a new index which has not been used in the past. Analysis considering the variable continuous will be reported along the optimal cut point value. The optimal cut point analysis is necessary to give a possible range of values where this index could be useful. $\endgroup$– GGACommented Aug 30, 2016 at 13:12
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
If you want to find "a possible range of values where this index could be useful" then you should examine the continuous relation of your index to outcome. Looking for a single cut point will not accomplish that task. See this question for discussion in the general regression context and this question in a broader machine-learning context.
For a novel index it's particularly important to understand how its values over its range are related to outcome. Maybe you will find that there is some upper or lower limit beyond which changes in the index don't matter, but you won't learn that unless you look in detail. It's also important to see whether the index adds anything to already established prognostic variables, something you can't do with a single-predictor Cox model.
Furthermore, the word "optimal" can hide a lot of assumptions. In the classification context, it can make the assumption that false positives have the same costs as false negatives, which isn't always the case. See this answer for discussion of cut points in the context of Cox models.
If you nevertheless are compelled to look for a cut point, your proposed method seems to be essentially that used by the cutp function in the R survMisc package. I would recommend that you try your cut point selection on multiple bootstrap samples from your data, as that best mimics repeated sampling from the underlying population. Unless your numeric index is hiding some true dichotomy in an underlying phenomenon, my hunch is that you will find a pretty wide range of "optimal" cut points, however defined, among those repeated samples. The bootstrap results at least will show your readers how much reliability to associate with the cut point value that you propose.
-
$\begingroup$ Thanks for the quality answer, I indeed used the function cutp with the lowest p value cutpoint for my analysis. So you suggest a bootstrap resampling and see how is the distribution of optimal cutpoints? What would be considered good enough for the resampling? My sample size is about 100 $\endgroup$– GGACommented Aug 30, 2016 at 16:50
-
1$\begingroup$ @GGA : A few hundred bootstrap samples (sampling the same number of cases, but with replacement for bootstrap) should show what's going on in terms of variability of cut points. I've used 1000 bootstraps with a 300-case situation, but that might be overkill. $\endgroup$– EdMCommented Aug 30, 2016 at 18:05