A week ago or so I was at a conference. Long story short, I ran into a friend who is quite good at machine learning so I asked them a question about why I might be getting what I think is poor fit on my GBM and randomForest models in R.
Unfortunately, I can't remember exact what they said - but I remember part of what they said, so I was hoping the community could help inform me better:
Basically, I have a 20+ year set of time series data set (sampled seasonally - 4 times/year over multiple species) with a fair number of predictors. I am trying to use this data for inference instead of prediction - I am not trying to forecast further into the time series, rather trying to explain an interesting phenomenom we are seeing throughout the time series.
Basically, they said that I am dividing my training and testing set wrong and this is why I have poor predictive preformance - because this is a time series I need to sample my data differently instead of just splitting it 80/20 or 70/30 randomly - I need to have each time point sampled equally, which I am interpreting as a rolling sample. I searched around the site and can't seem to find much (other than what is linked below). Can anyone elaborate on this?
I have run across this:
which provides a link to:
However, createTimeSlices doesn't seem to sample the data in different proportions and it seems to place ALL the data in the training and ALL the data in the testing set. At this point it is just confusing (example taken from:
library(caret) library(ggplot2) library(pls) data(economics) timeSlices <- createTimeSlices(1:nrow(economics), initialWindow = 36, horizon = 12, fixedWindow = TRUE) str(timeSlices,max.level = 1) List of 2 $ train:List of 527 .. [list output truncated] $ test :List of 527 .. [list output truncated]
I basically want to do this but randomly sample it for a time series (if this appropriate for time series):
library(caret) data(economics) set.seed(101) train = round(0.8*dim(economics), 0) train.index = sample(1:dim(economics), train, replace=FALSE) economics.train = economics[train.index,] economics.test = economics[-train.index,] str(economics.train) > str(economics.train) Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 459 obs. of 6 variables: str(economics.test) > str(economics.test) Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 115 obs. of 6 variables: