5
$\begingroup$

A week ago or so I was at a conference. Long story short, I ran into a friend who is quite good at machine learning so I asked them a question about why I might be getting what I think is poor fit on my GBM and randomForest models in R.

Unfortunately, I can't remember exact what they said - but I remember part of what they said, so I was hoping the community could help inform me better:

Basically, I have a 20+ year set of time series data set (sampled seasonally - 4 times/year over multiple species) with a fair number of predictors. I am trying to use this data for inference instead of prediction - I am not trying to forecast further into the time series, rather trying to explain an interesting phenomenom we are seeing throughout the time series.

Basically, they said that I am dividing my training and testing set wrong and this is why I have poor predictive preformance - because this is a time series I need to sample my data differently instead of just splitting it 80/20 or 70/30 randomly - I need to have each time point sampled equally, which I am interpreting as a rolling sample. I searched around the site and can't seem to find much (other than what is linked below). Can anyone elaborate on this?

I have run across this:

Multiclass SVM + Ineffective X Validation, Time Series Prediction

https://stackoverflow.com/questions/24758218/time-series-data-spliting-and-model-evaluation?lq=1

which provides a link to:

http://robjhyndman.com/hyndsight/crossvalidation/
http://robjhyndman.com/hyndsight/tscvexample/

However, createTimeSlices doesn't seem to sample the data in different proportions and it seems to place ALL the data in the training and ALL the data in the testing set. At this point it is just confusing (example taken from:

https://stackoverflow.com/questions/24758218/time-series-data-spliting-and-model-evaluation?lq=1).

library(caret)
library(ggplot2)
library(pls)

data(economics)

timeSlices <- createTimeSlices(1:nrow(economics), 
                            initialWindow = 36, horizon = 12, fixedWindow = TRUE)


str(timeSlices,max.level = 1)
List of 2
 $ train:List of 527
  .. [list output truncated]
 $ test :List of 527
  .. [list output truncated]

I basically want to do this but randomly sample it for a time series (if this appropriate for time series):

library(caret)
data(economics)
set.seed(101)
train = round(0.8*dim(economics)[1], 0)
train.index = sample(1:dim(economics)[1], train, replace=FALSE)
economics.train = economics[train.index,]
economics.test = economics[-train.index,]

str(economics.train)
> str(economics.train)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   459 obs. of  6 variables:

str(economics.test)
> str(economics.test)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   115 obs. of  6 variables:
Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I never work with time-series but it seems like you cannot split train/test using random select, why not getting last 5 years for testing and the other data for training? $\endgroup$
    – jrdi
    Commented Jul 7, 2016 at 15:42
  • 1
    $\begingroup$ Even if you don't want to forecast, your test data have to be after your training data. $\endgroup$
    – el Josso
    Commented Jul 7, 2016 at 15:50
  • $\begingroup$ @elJosso Why must that be the case? In principle, I don't see why a hypothesis generated from a training set can't be tested on data from an earlier time period. $\endgroup$
    – mkt
    Commented Jul 7, 2016 at 16:09
  • $\begingroup$ If you take your test before, I don't see what could go wrong. But if you take it between to training data, you can have problems. Even more if you want to do inference. $\endgroup$
    – el Josso
    Commented Jul 8, 2016 at 7:20

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.