I am trying to model ROC curves for a longitudinal dataset where participants were measured between 1-13 times. Time is not of interest but the fact that the measurements are autocorrelated an issue. I have a continuous variable and a y/n designation of disease or no disease and I want to subset my data and create different curves for boys, girls, age groups, and ethnic groups. Then I want to compare AUCs to see if the continuous measure is a better predictor of disease for girls in one ethnic group compared to another or for boys that are 11 compared to 16 years old, etc.

This paper seems to be the only piece of instructions I can find on the topic. The authors here generate a repeated measure logistic model, obtain estimated probability of positivity for each observation based on the model and form all discordant pairs, then calculate area under the ROC curve by Wilcoxon nop-parametric approach. The authors introduce this approach to evaluate the impact of co-variates on the potency and accuracy of a test or bio-marker, and I am not interested in evaluating covariates.

My questions are:

  1. Does this approach make sense when not including covariates; to model a logistic regression for repeated measures then calculate ROC curves based on the estimated probabilities?

  2. Could I use a linear regression with Cochrane-Orcutt to account for the autocorrelation in my dataset then use the predicted values of my continuous variable to model the ROC curves?

    2a. I am interested in this option because I also want the Youden's Index value from the ROC curve (which indicates the cutoff of the continuous variable with the optimal tradeoff between sensitivity and specificity) and if I model probabilities instead of predicted values then I don't know how to get Youden's Index.

  3. By some miracle does anyone have experience / syntax / code to do this in R or SPSS? The authors in the above example provide SAS code but I do not have easy access to SAS but am quite proficient in SPSS and fairly proficient in R. Thanks!

  • $\begingroup$ Your problem is not the roc analysis, but your predictor. You need to define what is a "positive" prediction. Does any measurement above a cut-off define the sample as positive? Or are you looking for a trend? Once you've defined this the ROC part will follow naturally $\endgroup$ – Calimo Oct 24 '16 at 7:39
  • $\begingroup$ Thanks @Calimo, but I am not sure I understand. I have a definition of disease, with yes / no for each individual. I want to evaluate how a continuous predictor discriminates between diseased and not diseased, compare across groups to see if this predictor is better or worse in certain cases. Are you saying I need to define a cut-off for "positive" prediction in my continuous predictor variable as well as in my definition of disease variable? $\endgroup$ – Jennifer McConnell Oct 25 '16 at 20:03
  • $\begingroup$ ROC analysis will define the cut-off. What you need to figure out is what happens to the patient when the measurement disagree and change over time. $\endgroup$ – Calimo Oct 25 '16 at 20:28
  • $\begingroup$ If you want to be conservative (and not make full use of the data) you could sub-sample a single observation per person (and perhaps repeat over and over). $\endgroup$ – scottyaz Feb 12 '18 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.