# Determining the winner model from ROC curve

I have a ROC curve for a specific hyperparameter tuning setting for a decision tree. The candidate values for which I plot are 0.1, 0.01, 0.001, 0.0001. I want to determine (visually) which model has the highest sensitivity given a specificity of 67%.

library(tidyverse)
library(tidymodels)

set.seed(123)
tuning_folds <- vfold_cv(so, v = 5, strata = "remote")

fit_res <- decision_tree(cost_complexity = tune()) %>%
set_engine("rpart") %>%
set_mode("classification") %>%
tune_grid(
remote ~ .,
resamples = tuning_folds,
grid = tibble(cost_complexity = c(0.1, 0.01, 0.001, 0.0001)),
control = control_grid(save_pred = TRUE)
)

fit_res %>%
pull(.predictions) %>%
bind_rows() %>%
group_by(cost_complexity) %>%
roc_curve(truth = remote, .pred_Remote) %>%
ungroup() %>%
ggplot(aes(
x = 1 - specificity,
y = sensitivity,
color = as.factor(cost_complexity))
) +
geom_line() +
theme_bw()


here is my plot

> dput(head(so))
structure(list(country = structure(c(5L, 5L, 4L, 4L, 5L, 5L), .Label = c("Canada",
"Germany", "India", "United Kingdom", "United States"), class = "factor"),
salary = c(63750, 93000, 40625, 45000, 1e+05, 170000), years_coded_job = c(4L,
9L, 8L, 3L, 8L, 12L), open_source = c(0, 1, 1, 1, 0, 1),
hobby = c(1, 1, 1, 0, 1, 1), company_size_number = c(20,
1000, 10000, 1, 10, 100), remote = structure(c(1L, 1L, 1L,
1L, 1L, 1L), .Label = c("Remote", "Not remote"), class = "factor"),
career_satisfaction = c(8L, 8L, 5L, 10L, 8L, 10L), data_scientist = c(0,
0, 1, 0, 0, 0), database_administrator = c(1, 0, 1, 0, 0,
0), desktop_applications_developer = c(1, 0, 1, 0, 0, 0),
developer_with_stats_math_background = c(0, 0, 0, 0, 0, 0
), dev_ops = c(0, 0, 0, 0, 0, 1), embedded_developer = c(0,
0, 0, 0, 0, 0), graphic_designer = c(0, 0, 0, 0, 0, 0), graphics_programming = c(0,
0, 0, 0, 0, 0), machine_learning_specialist = c(0, 0, 0,
0, 0, 0), mobile_developer = c(0, 1, 0, 0, 1, 0), quality_assurance_engineer = c(0,
0, 0, 0, 0, 0), systems_administrator = c(1, 0, 1, 0, 0,
1), web_developer = c(0, 0, 0, 1, 1, 1)), row.names = c(NA,
-6L), class = c("tbl_df", "tbl", "data.frame"))


I cannot make a clear-cut distinction as to which model with the orange(1e-04) or green (0.001) or both of them wins with a specificity of 67%.

• Go to $x=0.33$ (specificity of $0.67$, draw a vertical line, and pick the curve with the highest $y$-value, same as you would do to see which of any other collection of functions has the highest $y$-value for a given $x$-value. Is there something about this that does not work? (It looks like the blue $0.01$ hyperparameter is your winner.) // Why are some of your curves decreasing? The ROC curves can be constant, but they cannot decrease. stats.stackexchange.com/questions/189494/… – Dave May 26 at 15:55
• ROC curves are non-decreasing, but I see spots where the curve declines, so I think there is some problem with how the ROC was constructed. – Sycorax May 26 at 16:04

Now I see your issue. Remember that the $$x$$-axis is $$1-\text{specificity}$$, not $$\text{specificity}$$ itself. You should be looking at $$x=0.33$$, not $$x=0.67$$, as the $$\text{sensitivity}=0.67$$ corresponds to $$x=1-\text{sensitivity}=0.33$$.
Then you draw a vertical line with the geom_vline(xintercept = 0.33) command and see which curve has the highest value. Since the blue curve with the $$0.01$$ hyperparameter is the highest from about $$x=0.25$$ to $$x=0.50$$, $$0.01$$ will be your winning hyperparameter.
• Your edit does not alter the approach to finding the winning hyperparameter. If you have questions about debugging your ROC curve code, consider posting at Stack Overflow or using the pROC package. – Dave May 26 at 16:05