Integrated biomarkers for ROC

Question

I'm am looking for ways to combine multiple quantitative values in order to build a ROC curve with specificity and sensitivity. This seems to be common in multiple biomarkers paper, but I can't find any direct way to do it. CombiROC seemed to be made for this, but the Shiny app is no longer working and the packages haven't been updated, which makes me question its use.

I have multiple protein concentrations for a given timepoint (no follow-up) from a cohort of patients with binary presentation (could be Control vs diseased). Using these proteins alone, I can build a ROC curve. I found that some to be associated with my disease. I would like to push my analysis further to have a model where I can combine/integrate 2 or 3 of these markers so refine the diagnostic and have better AUC/Spe/Sen values. For instance, at a certain point, having both A and B elevated could have a worse prognostic compared to just high A.

So far, this is the way I found to do it: I first build a glm model, from which I predict the chance of falling into the group. I finally calculate the ROC by plotting the predicted vs actual outcome.

#Build a glm, with the clinical group (binairy outcome)
Combined.glm <- glm(dataset$Group1 ~ 
                               dataset$Analyte1 +
                               dataset$Analyte2 +
                               dataset$Analyte3, 
                               family = "binomial")

#Use the model to predict the group outcome
dataset$prob.Group1=predict(Combined.glm,type=c("response"))

#Build the ROC
roc<- roc(dataset$Group1, dataset$prob.Group1)

I would like to know if this is a correct way to do this, or if not, how should do this?

Thanks!

Edit: added clarifications and rephrased

Answer 1

The ROC curve, specificity and sensitivity shouldn't be primary concerns. You need a well-calibrated model of the probability of disease status based on a set of predictors. Once that's available, the ROC curve might not be helpful at all.

You've made a start at that with your binomial generalized linear model, but only a start. Your model implicitly assumes a linear association of each of your analytes with the log-odds of disease status, each independent of the values of the others. That's a pretty strong assumption.

There is extensive information on this site about how to set up regression models in general and about issues specific to binary regression, available from the search function and tags. Frank Harrell's Regression Modeling Strategies is a freely available central resource with Chapters 10, 11 and 12 devoted specifically to binary logistic regression. Follow those guidelines to construct a well-calibrated model that doesn't overfit the data.

Once you have that model, you have pretty much all that you need. It tells you the log-odds of disease based on the analyte levels. That's a quantitative form of "Your A and B proteins are elevated therefore, you are more at risk of Z complication"; the regression model provides information on how much more likely disease is based on how much the proteins are elevated.

Furthermore, the probability model allows a cost-based choice of probability cutoff for classification, if needed, depending on the relative costs of false-positive and false-negative class assignments. Quoting from this page:

With 0 cost of true classification, say that the costs of false positives and false negatives are scaled to sum to 1. Call the cost of a false positive $c$ so that the cost of a false negative is $(1-c)$. Then the optimal probability classification cutoff for minimizing expected cost is at $c$.

You don't need an ROC curve to make that choice for classification. Yes, you can think of that choice as moving along the ROC curve with its associated values of sensitivity and specificity as a function of the model's linear-predictor values. But that ROC curve doesn't directly illustrate the disease probability as a function of the linear predictor.

If you want to show a plot, show plots of how the log-odds of the outcome changes as a continuous function of each analyte value, similar to Figure 11.2 of Regression Modeling Strategies. Show a plot documenting the model's calibration, with good agreement between predicted and observed outcomes, as in Figure 11.5.

Similarly, a choice of probability cutoff will have associated values of specificity and sensitivity, but they aren't then of primary concern. The sensitivity/specificity tradeoff should be based on relative costs, for which the probability model is most directly useful.

If you nevertheless want to display a ROC curve, the correct way to do so would depend on the documentation for the roc() function, whose package you don't specify. On this statistics web site, software-specific questions are considered off-topic, anyway. If that gives a plot of sensitivity versus 1-specificity then you have an ROC curve. I'd recommend further adorning the curve with associated probabilities and/or linear-predictor values along its length, as those are what's of most general interest and importance.

Added to respond to comment

I want to see if using more than one can bring a more precise diagnostic than one alone.

Frank Harrell discusses this in this web post. A likelihood-ratio test comparing a model with the original and the additional predictors to that with only the original predictor can gauge the "statistical significance" of the additional predictors. A related "adequacy index" describes the adequacy of the original model without the additional predictors. He illustrates the approach with a logistic regression model.