Determining the winner model from ROC curve

Question

I have a ROC curve for a specific hyperparameter tuning setting for a decision tree. The candidate values for which I plot are 0.1, 0.01, 0.001, 0.0001. I want to determine (visually) which model has the highest sensitivity given a specificity of 67%.

library(tidyverse)
library(tidymodels)

so <- read_rds("stackoverflow.rds")

set.seed(123)
tuning_folds <- vfold_cv(so, v = 5, strata = "remote")

fit_res <- decision_tree(cost_complexity = tune()) %>%
  set_engine("rpart") %>%
  set_mode("classification") %>%
  tune_grid(
    remote ~ .,
    resamples = tuning_folds,
    grid = tibble(cost_complexity = c(0.1, 0.01, 0.001, 0.0001)),
    control = control_grid(save_pred = TRUE)
  )

fit_res %>%
  pull(.predictions) %>%
  bind_rows() %>%
  group_by(cost_complexity) %>%
  roc_curve(truth = remote, .pred_Remote) %>%
  ungroup() %>%
  ggplot(aes(
    x = 1 - specificity,
    y = sensitivity,
    color = as.factor(cost_complexity))
  ) +
  geom_line() +
  theme_bw()

here is my plot

> dput(head(so))
structure(list(country = structure(c(5L, 5L, 4L, 4L, 5L, 5L), .Label = c("Canada", 
"Germany", "India", "United Kingdom", "United States"), class = "factor"), 
    salary = c(63750, 93000, 40625, 45000, 1e+05, 170000), years_coded_job = c(4L, 
    9L, 8L, 3L, 8L, 12L), open_source = c(0, 1, 1, 1, 0, 1), 
    hobby = c(1, 1, 1, 0, 1, 1), company_size_number = c(20, 
    1000, 10000, 1, 10, 100), remote = structure(c(1L, 1L, 1L, 
    1L, 1L, 1L), .Label = c("Remote", "Not remote"), class = "factor"), 
    career_satisfaction = c(8L, 8L, 5L, 10L, 8L, 10L), data_scientist = c(0, 
    0, 1, 0, 0, 0), database_administrator = c(1, 0, 1, 0, 0, 
    0), desktop_applications_developer = c(1, 0, 1, 0, 0, 0), 
    developer_with_stats_math_background = c(0, 0, 0, 0, 0, 0
    ), dev_ops = c(0, 0, 0, 0, 0, 1), embedded_developer = c(0, 
    0, 0, 0, 0, 0), graphic_designer = c(0, 0, 0, 0, 0, 0), graphics_programming = c(0, 
    0, 0, 0, 0, 0), machine_learning_specialist = c(0, 0, 0, 
    0, 0, 0), mobile_developer = c(0, 1, 0, 0, 1, 0), quality_assurance_engineer = c(0, 
    0, 0, 0, 0, 0), systems_administrator = c(1, 0, 1, 0, 0, 
    1), web_developer = c(0, 0, 0, 1, 1, 1)), row.names = c(NA, 
-6L), class = c("tbl_df", "tbl", "data.frame"))

I cannot make a clear-cut distinction as to which model with the orange(1e-04) or green (0.001) or both of them wins with a specificity of 67%.

Answer 1

Now I see your issue. Remember that the $x$-axis is $1-\text{specificity}$, not $\text{specificity}$ itself. You should be looking at $x=0.33$, not $x=0.67$, as the $\text{sensitivity}=0.67$ corresponds to $x=1-\text{sensitivity}=0.33$.

Then you draw a vertical line with the geom_vline(xintercept = 0.33) command and see which curve has the highest value. Since the blue curve with the $0.01$ hyperparameter is the highest from about $x=0.25$ to $x=0.50$, $0.01$ will be your winning hyperparameter.