2
$\begingroup$

I have a ROC curve for a specific hyperparameter tuning setting for a decision tree. The candidate values for which I plot are 0.1, 0.01, 0.001, 0.0001. I want to determine (visually) which model has the highest sensitivity given a specificity of 67%.

library(tidyverse)
library(tidymodels)

so <- read_rds("stackoverflow.rds")

set.seed(123)
tuning_folds <- vfold_cv(so, v = 5, strata = "remote")

fit_res <- decision_tree(cost_complexity = tune()) %>%
  set_engine("rpart") %>%
  set_mode("classification") %>%
  tune_grid(
    remote ~ .,
    resamples = tuning_folds,
    grid = tibble(cost_complexity = c(0.1, 0.01, 0.001, 0.0001)),
    control = control_grid(save_pred = TRUE)
  )

fit_res %>%
  pull(.predictions) %>%
  bind_rows() %>%
  group_by(cost_complexity) %>%
  roc_curve(truth = remote, .pred_Remote) %>%
  ungroup() %>%
  ggplot(aes(
    x = 1 - specificity,
    y = sensitivity,
    color = as.factor(cost_complexity))
  ) +
  geom_line() +
  theme_bw()

here is my plot

ROC

> dput(head(so))
structure(list(country = structure(c(5L, 5L, 4L, 4L, 5L, 5L), .Label = c("Canada", 
"Germany", "India", "United Kingdom", "United States"), class = "factor"), 
    salary = c(63750, 93000, 40625, 45000, 1e+05, 170000), years_coded_job = c(4L, 
    9L, 8L, 3L, 8L, 12L), open_source = c(0, 1, 1, 1, 0, 1), 
    hobby = c(1, 1, 1, 0, 1, 1), company_size_number = c(20, 
    1000, 10000, 1, 10, 100), remote = structure(c(1L, 1L, 1L, 
    1L, 1L, 1L), .Label = c("Remote", "Not remote"), class = "factor"), 
    career_satisfaction = c(8L, 8L, 5L, 10L, 8L, 10L), data_scientist = c(0, 
    0, 1, 0, 0, 0), database_administrator = c(1, 0, 1, 0, 0, 
    0), desktop_applications_developer = c(1, 0, 1, 0, 0, 0), 
    developer_with_stats_math_background = c(0, 0, 0, 0, 0, 0
    ), dev_ops = c(0, 0, 0, 0, 0, 1), embedded_developer = c(0, 
    0, 0, 0, 0, 0), graphic_designer = c(0, 0, 0, 0, 0, 0), graphics_programming = c(0, 
    0, 0, 0, 0, 0), machine_learning_specialist = c(0, 0, 0, 
    0, 0, 0), mobile_developer = c(0, 1, 0, 0, 1, 0), quality_assurance_engineer = c(0, 
    0, 0, 0, 0, 0), systems_administrator = c(1, 0, 1, 0, 0, 
    1), web_developer = c(0, 0, 0, 1, 1, 1)), row.names = c(NA, 
-6L), class = c("tbl_df", "tbl", "data.frame"))

I cannot make a clear-cut distinction as to which model with the orange(1e-04) or green (0.001) or both of them wins with a specificity of 67%.

$\endgroup$
2
  • 1
    $\begingroup$ Go to $x=0.33$ (specificity of $0.67$, draw a vertical line, and pick the curve with the highest $y$-value, same as you would do to see which of any other collection of functions has the highest $y$-value for a given $x$-value. Is there something about this that does not work? (It looks like the blue $0.01$ hyperparameter is your winner.) // Why are some of your curves decreasing? The ROC curves can be constant, but they cannot decrease. stats.stackexchange.com/questions/189494/… $\endgroup$ – Dave May 26 at 15:55
  • 1
    $\begingroup$ ROC curves are non-decreasing, but I see spots where the curve declines, so I think there is some problem with how the ROC was constructed. $\endgroup$ – Sycorax May 26 at 16:04
2
$\begingroup$

Now I see your issue. Remember that the $x$-axis is $1-\text{specificity}$, not $\text{specificity}$ itself. You should be looking at $x=0.33$, not $x=0.67$, as the $\text{sensitivity}=0.67$ corresponds to $x=1-\text{sensitivity}=0.33$.

Then you draw a vertical line with the geom_vline(xintercept = 0.33) command and see which curve has the highest value. Since the blue curve with the $0.01$ hyperparameter is the highest from about $x=0.25$ to $x=0.50$, $0.01$ will be your winning hyperparameter.

$\endgroup$
3
  • 1
    $\begingroup$ I still wonder why your ROC curves have decreasing sections, though. Consider checking and debugging the code you used to calculate the sensitivities and specificities. $\endgroup$ – Dave May 26 at 16:03
  • $\begingroup$ I have updated my question and also an MRE example of my data. Maybe it helps more to investigate why they are decreasing. $\endgroup$ – Ranji Raj May 26 at 16:03
  • $\begingroup$ Your edit does not alter the approach to finding the winning hyperparameter. If you have questions about debugging your ROC curve code, consider posting at Stack Overflow or using the pROC package. $\endgroup$ – Dave May 26 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.