Proper way to finding area under ROC curve (AUC)

Question

I have data of 200 persons with 2 variables:

1. C_level: Blood level of a chemical C (a numeric value)
2. Disease: Yes or No

I want to know if C_level can be used to predict the presence of Disease and what is the area under ROC curve. So Disease is y and C_level is X. However, I am not clear about proper method to achieve this. I see following options:

1. To take Disease variable as y_true and C_level as X or y_score and create an ROC curve with some software like: roc_auc_score function using code similar to following:

   AUC = roc_auc_score(y, X, multi_class='ovr')

2. Take the data; fit a classifier (e.g. Logistic regression) and find AUC of ROC curve with predicted probabilities of X using code similar to following (from here):

   clf = LogisticRegression(solver="liblinear").fit(X, y)

   AUC = roc_auc_score(y, clf.predict_proba(X), multi_class='ovr')

3. Take the data; split it in train and test parts; fit a classifier (e.g. Logistic regression) with train part of data; and finally find AUC of ROC curve with test parts of data with code as in option 2.

Which of these is the correct method for finding area under the ROC curve (AUC)?

Edit: I found another option:

4. Get cross validation score as given here. However, I am not sure the value given by cross_val_score function is same as area under ROC curve.

Answer 1

The three options you outline differ in what they measure.

Option (1) measures the extent to which a higher C-level is associated with disease.

Option (2) will be the same as (1) when the models are monotonic transformations C-level that don't reverse sign. For example, if the model for the log-odds of disease is $0.4 + 2.1 \text{C-level}$, this model will have the same AUC as (1) because it only shifts and re-scales C-level. The AUC is a statistic of ranks, so shifting and positive re-scaling doesn't change the AUC.

On the other hand, a model that is not monotonic will change the AUC because the ranks are changed. And likewise, negative rescaling will also change the AUC because the ranks are reversed. (This is what OP discovered when they changed how gender was coded in this question Validity of AUC for binary categorical variables)

Option (3) is one way of assessing if (2) has discovered a spurious model.

Answer 2

It seems like you are trying to obtain an estimate for the discriminative capability of a logistic regression model. Let me propose a fifth method:

5. The optimism corrected bootstrap estimate of the AUC for your model.

The optimism bootstrap corrected estimate is discussed in the following books: Elements of Statistical Learning, Regression Modelling Strategies, and Clinical Prediction Models. For a review, I suggest you read these.

As compared to the methods you describe, the optimism bootstrap has the benefit of using all the data to estimate the AUC and using all the data to estimate the optimism. No data is held out per se as would be in cross validation. Because you're using python, I will point you to my blog post on the optimism bootstrap here.