Getting ROC curve from markov chain in r

Question

I've been working on some attribution modeling in R, following the markov chain-based methodology described here.

In order to compare the sensitivity and specificity trade-offs of using models of different markov orders, I'd like to look at a ROC curve. I do not know how to technically implement this, using the dataset produced by the channelattribution and markovchain packages. Specifically, I do not know how to produce a prediction and label dataset to feed into ROCR package. Any pointers in the right direction would be greatly appreciated!

For reference, below is sample code for creating the first order markov model using a randomly generated dataset:

# simulating the "real" data
set.seed(354)
df2 <- data.frame(client_id = sample(c(1:1000), 5000, replace = TRUE),
date = sample(c(1:32), 5000, replace = TRUE),
channel = sample(c(0:9), 5000, replace = TRUE,
prob = c(0.1, 0.15, 0.05, 0.07, 0.11, 0.07, 0.13, 0.1, 0.06, 0.16)))
df2$date <- as.Date(df2$date, origin = "2015-01-01")
df2$channel <- paste0('channel_', df2$channel)

# aggregating channels to the paths for each customer
df2 <- df2 %>%
group_by(client_id) %>%
summarise(path = paste(channel, collapse = ' > '),
# assume that all paths were finished with conversion
conv = 1,
conv_null = 0) %>%
ungroup()

# calculating the models (Markov and heuristics)
mod2 <- markov_model(df2,
var_path = 'path',
var_conv = 'conv',
var_null = 'conv_null',
out_more = TRUE)`

Answer 1

I have the same question and got here by searching for some guidance. Here is how I thought one could produce predictions and ROC curves (but I expect the community to correct me if i'm going wrong).

The markov model would output the attribution weights of each channel which are basically conversion probabilities. Now say you have a customer who went through 3 different channels - take those channels probabilites and multiply them and you get the overall probability that this specific channel journey would result into a conversion.

Specify a threshold over which you would consider a probability of converstion to be "positive".

Do this for different thresholds and plot out the ROC curve.