I have a ROC curve for a specific hyperparameter tuning setting for a decision tree. The candidate values for which I plot are 0.1, 0.01, 0.001, 0.0001. I want to determine (visually) which model has the highest sensitivity given a specificity of 67%.
library(tidyverse)
library(tidymodels)
so <- read_rds("stackoverflow.rds")
set.seed(123)
tuning_folds <- vfold_cv(so, v = 5, strata = "remote")
fit_res <- decision_tree(cost_complexity = tune()) %>%
set_engine("rpart") %>%
set_mode("classification") %>%
tune_grid(
remote ~ .,
resamples = tuning_folds,
grid = tibble(cost_complexity = c(0.1, 0.01, 0.001, 0.0001)),
control = control_grid(save_pred = TRUE)
)
fit_res %>%
pull(.predictions) %>%
bind_rows() %>%
group_by(cost_complexity) %>%
roc_curve(truth = remote, .pred_Remote) %>%
ungroup() %>%
ggplot(aes(
x = 1 - specificity,
y = sensitivity,
color = as.factor(cost_complexity))
) +
geom_line() +
theme_bw()
here is my plot
> dput(head(so))
structure(list(country = structure(c(5L, 5L, 4L, 4L, 5L, 5L), .Label = c("Canada",
"Germany", "India", "United Kingdom", "United States"), class = "factor"),
salary = c(63750, 93000, 40625, 45000, 1e+05, 170000), years_coded_job = c(4L,
9L, 8L, 3L, 8L, 12L), open_source = c(0, 1, 1, 1, 0, 1),
hobby = c(1, 1, 1, 0, 1, 1), company_size_number = c(20,
1000, 10000, 1, 10, 100), remote = structure(c(1L, 1L, 1L,
1L, 1L, 1L), .Label = c("Remote", "Not remote"), class = "factor"),
career_satisfaction = c(8L, 8L, 5L, 10L, 8L, 10L), data_scientist = c(0,
0, 1, 0, 0, 0), database_administrator = c(1, 0, 1, 0, 0,
0), desktop_applications_developer = c(1, 0, 1, 0, 0, 0),
developer_with_stats_math_background = c(0, 0, 0, 0, 0, 0
), dev_ops = c(0, 0, 0, 0, 0, 1), embedded_developer = c(0,
0, 0, 0, 0, 0), graphic_designer = c(0, 0, 0, 0, 0, 0), graphics_programming = c(0,
0, 0, 0, 0, 0), machine_learning_specialist = c(0, 0, 0,
0, 0, 0), mobile_developer = c(0, 1, 0, 0, 1, 0), quality_assurance_engineer = c(0,
0, 0, 0, 0, 0), systems_administrator = c(1, 0, 1, 0, 0,
1), web_developer = c(0, 0, 0, 1, 1, 1)), row.names = c(NA,
-6L), class = c("tbl_df", "tbl", "data.frame"))
I cannot make a clear-cut distinction as to which model with the orange(1e-04) or green (0.001) or both of them wins with a specificity of 67%.