I'm confused about the resampling statistics obtained via the train() and trainControl() functions in R's Caret package. My goal is to perform cross-validation on a machine learning model for a time series by splitting the series into adjacent training/testing sets, ultimately enabling me to tune individual models and compare the different models I create.

I also would like to access the model results for the individual hold out sets (that is, the testing windows during cross validation) - which is what I thought Caret saved in the model$resample object. However, when I look closely at that object, it seems that Caret is saving the in-sample results, possibly from the final model (trained on the entire data set) with predictions corresponding to the indexes of the individual hold-out sets.

My questions are:

  1. Is this (the saving of in-sample rather than hold-out results) the intended behavior of the package?
  2. If so, is there a way to access the hold-out results I am looking for?

Here is an example (apologies for the verbosity):

## Libraries

## Data Acquisition and Wrangling

# download data from yahoo/google
getSymbols("^GSPC", from="2010-01-01")

# create returns series and some indicators
ret <- Delt(Op(GSPC), Cl(GSPC)) #day session return
atr <- ATR(GSPC[, c("GSPC.High", "GSPC.Low", "GSPC.Close")], 14)$atr
rsi <- RSI(Cl(GSPC), n=14, maType="SMA") #relative strength index
cmo <- CMO(Cl(GSPC)) #Chande momentm oscillator
dpo <- DPO(Cl(GSPC)) #de-trended price oscillator

dat <- cbind(ret, Lag(atr), Lag(rsi), Lag(cmo), Lag(dpo))
colnames(dat) <- c("ret", "atr", "rsi", "cmo", "dpo")

dat <- as.data.frame(dat[complete.cases(dat), ]) 

## Set up Time Series Cross Validation Windows

# create indexes for TSCV windows
init = 200 #initial window 
horiz = 20 #prediction horizon 
wdw <- createTimeSlices(1:nrow(dat), initialWindow = init, horizon = horiz, 
                        skip = horiz-1, fixedWindow = TRUE)

trainSlices <- wdw[[1]]
testSlices <- wdw[[2]]

# verify visually correct window setup:

# custom summary function
absretSummary <- function (data, lev = NULL, model = NULL) {
  positions <- sign(data[, "pred"])
  profits <- positions*data[, "obs"]
  profit <- prod(1+profits)-1
  names(profit) <- 'profit'

# trainControl function
cntrl <- trainControl(savePredictions=FALSE, returnResamp="all", index=trainSlices, indexOut=testSlices, summaryFunction=absretSummary)   

# train gbm
gbm.model <- train(x=dat[, -1], y=dat[, 1], method="gbm", trControl=cntrl,  
                   maximize=TRUE, metric="profit", tuneGrid = expand.grid(interaction.depth = c(1,2),
                                                         n.trees = c(50,100),
                                                         shrinkage = c(0.1),
                                                         n.minobsinnode = c(10)))

gbm.model$resample[, 1] # no negative returns! Has to be in-sample!

I looked extensively into the Caret GitHub repo and couldn't find anything that explained the behavior I described above. So I wrote some code to perform the same analysis without using any of Caret's helper functions. And I got the exact same result. Therefore, the unexpected behavior that I described has a different source than the one I implied in my question, possibly some future-peeking in my slicing and dicing of the time series - although I haven't been able to figure this out yet.

I think this question should be closed and a different one asked, possibly on another exchange.

| cite | improve this answer | |
  • $\begingroup$ You can close the question yourself. $\endgroup$ – SmallChess Apr 3 '17 at 7:34
  • $\begingroup$ Do I need a certain reputation for that? I don't see the 'close' link referred to in the help documentation. $\endgroup$ – ManChild Apr 3 '17 at 12:06
  • $\begingroup$ Ask on stats.meta.stackexchange.com. I don't know why. $\endgroup$ – SmallChess Apr 3 '17 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.