2
$\begingroup$

I have this model from the uplift package,

mod.RF <- upliftRF(as.numeric(response) ~.
                   +trt(group), 
                   data = df,
                   mtry = 3,
                   ntree = 100,
                   split_method = "KL",
                   minsplit = 200,
                   interaction.depth = 1, #also try with id = 2
                   verbose = TRUE)

I want to make predictions and evaluate those predictions. I'd like to create a test set with 10-fold cross validation. I know how to do that in caret when that it being used to make a model.

How do I evaluate the tree using 10-fold cross validation outside of caret?

Put another way - how can I make new data that I can fill in for the newdata object in predict()and can I use cross validation to do that?

$\endgroup$
1
  • $\begingroup$ Not asked, but a relevant addition given the question title: a good metric to use for cross-validating uplift models is the transformed outcome loss T*Y/p(T) - (1-T)Y(1-p(T)), where T is a binary indicator if the individual received treatment. A nice explanation is given in Hitsch, G. J., & Misra, S. (2018). Heterogeneous Treatment Effects and Optimal Targeting Policy Evaluation. SSRN. $\endgroup$
    – Johannes
    Oct 3, 2019 at 19:26

1 Answer 1

0
$\begingroup$

This model is not a decision tree, it is a random forest with ntree = 100 trees.

Here is one general way to do cross validation with the modelr package.

library("tidyverse")
library("modelr")
library("uplift")

# Use example in upliftRF's documentation
set.seed(123)
dd <- sim_pte(n = 1000, p = 20, rho = 0, sigma =  sqrt(2), beta.den = 4)
dd$treat <- ifelse(dd$treat == 1, 1, 0)

### fit uplift random forest

uplift_model <- function(data) {
  # It is easier to write a separte "fit" function when
  # there are many parameters
  upliftRF(y ~ X1 + X2 + X3 + X4 + X5 + X6 + trt(treat),
           data = data,
           mtry = 3,
           ntree = 100,
           split_method = "KL",
           minsplit = 200,
           verbose = TRUE)
}
predict_uplift <- function(mod, newdata) {
  # For some reason, necessary with `predict.upliftRF`
  newdata <- as_tibble(newdata)
  predict(mod, newdata)
}

# 5-fold cross validation
folds <- crossv_kfold(dd, 3)

folds %>%
  mutate(model = map(train, uplift_model)) %>%
  mutate(pred = map2(model, test, predict_uplift))
#> # A tibble: 3 x 5
#>   train          test           .id   model          pred           
#>   <list>         <list>         <chr> <list>         <list>         
#> 1 <S3: resample> <S3: resample> 1     <S3: upliftRF> <dbl [334 x 2]>
#> 2 <S3: resample> <S3: resample> 2     <S3: upliftRF> <dbl [333 x 2]>
#> 3 <S3: resample> <S3: resample> 3     <S3: upliftRF> <dbl [333 x 2]>

Created on 2019-03-28 by the reprex package (v0.2.1)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.