0
$\begingroup$

based on code presented in thread: How to find a GBM Prediction Interval

I am trying to apply this to my dataset. Below is my full code, and I am having issues with the bootstrap function.

library(caret)
require(foreign)

set.seed(825)
Ridership <- read.spss("V:/Metro/Coverage/ROUTE_MODEL2.sav",use.value.labels=TRUE, to.data.frame = TRUE)

set.seed(825)
fitControl <- trainControl(method = "cv", number = 2)
gbmGrid <-  expand.grid(interaction.depth = (20:21), n.trees = (750), shrinkage = c(0.07))
x <- Ridership[, -148]
y <- Ridership[, 148]

gbmFit <- train(x=x,y=y,"gbm", tuneGrid = gbmGrid, n.minobsinnode = 2, trControl =fitControl, verbose=FALSE)
gbmFit
x.pt <- quantile(Ridership$TOT_RIDERSHIP, c(0.25, 0.5, 0.75))
p <- plot(gbmFit, newdata = Ridership[, -148], grid.levels = x.pt, return.grid = TRUE)
p
library(boot)
bootfun <- function(data, indices) {
  data <- data[indices,]
  x <- Ridership[, -148]
  y <- Ridership[, 148]
  gbmFit <- train(x=x,y=y,"gbm", tuneGrid = gbmGrid, n.minobsinnode = 2, trControl =fitControl, verbose=FALSE)
  plot(gbmFit, newdata = Ridership[, -148], grid.levels = x.pt, return.grid = TRUE)$y
}
b <- boot(data = Ridership, statistic = bootfun, R = 5) 
lims <- t(apply(b$t, 2, FUN = function(x) quantile(x, c(0.025, 0.975))))

When I run the code, the lim(only show 1, and nothing more. I am not exactly sure what to define in the Bootstrap function. I have flipped through the bootstrap package code, but it still is not clear to me what I am doing wrong. Thanks in advance!

$\endgroup$
  • $\begingroup$ What happens if you replace gbmFit with gbmFit$finalModel in your call to plot.gbm? $\endgroup$ – ErikL Sep 30 '14 at 18:15
  • $\begingroup$ I get some output. It shows the 2.5 and 97.5 numbers, but there are 100 points (there are only 75 data points in the dataset), and there are only 4 values, sets 1:50 are the same and sets 51:100 are the same. Not what I was expecting! $\endgroup$ – CooperBuckeye05 Sep 30 '14 at 19:09
  • $\begingroup$ Started to write a comment but I'll write an answer instead... $\endgroup$ – ErikL Oct 1 '14 at 6:37
1
$\begingroup$

Hard to say without a reproducible example but some pointers based on what I can understand from the code:

  • For plot.gbm you need to pass a gbm object as well as the variable to plot. In your case something like p <- plot(gbmFit$finalModel, i.var = ..., grid.levels = x.pt, ...), where i.var is the variable you want partial dependencies for. The length defaults to 100, see ?plot.gbm and I think that is why you get 100 points. The presence of grid.levels should override this if plot.gbm is called correctly.
  • 2-fold CV is not likely to give a good estimate of performance, and with 75 data points I would use the bootstrap (or maybe leave one out CV) and restrict the trees to be of depth 1--3 (or so) unless you have very strong prior knowledge that you require depth 20 or 21 trees.

The book Applied Predictive Modeling by Max Kuhn and Kjell Johnson is a great go-to source for tuning gbm ensembles (and tuning predictive models in general).

Also note that this code gives confidence intervals for predicted values instead of prediction intervals as pointed out by the comments in the above mentioned thread.

Hope this helps!

$\endgroup$
  • $\begingroup$ Thank you for your help! In this case, do I use the name of my dependent var, or do I use the number of indices? If I do use the name of the dependent var what is the proper way to call it? The literature I have found on i.var is not too specific on how to use it. Thanks again! $\endgroup$ – CooperBuckeye05 Oct 1 '14 at 12:30
  • $\begingroup$ According to the documentation you can do both (and I think you mean the _in_dependent variable?) Both plot.gbm(gbmFit$finalModel, i.var = 5, ...) and plot.gbm(gbmFit$finalModel, i.var = "variableName", ...) should work if the 5th variable is called "variableName". I just use the variable name as I find it easier to keep track of what's going on. If you do not specify a grid.levels = you'll get 100 points spread evenly across the distribution of the variable and return.grid = TRUE returns a data set. Play around with the different arguments and see what works best for your purpose. $\endgroup$ – ErikL Oct 5 '14 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.