3
$\begingroup$

I have a monthly sales data set from 2018 January onwards. I would like to know from expert what is the optimum train test split and minimum train test split. Also to mention that my data includes 2020 year data where the sales have been influenced due to pandemic, and 2021 have been recovering year.

$\endgroup$
1
  • 1
    $\begingroup$ The word "optimal" suggests that you have some figure of merit that you're using to compare alternative split methods, but you do not mention any details about this in your post. Closely related, possibly a duplicate: stats.stackexchange.com/questions/14099/… $\endgroup$
    – Sycorax
    Sep 22 at 13:25
9
$\begingroup$

I am not aware of any theoretical work giving minimum or optimum train/test set splits for time series, and I doubt that such general guidelines could be given with any theoretical foundations.

Sales data frequently exhibit seasonality. So one reasonable split would be to hold out the entire last year as a test set, which still gives you more than two years to fit your models on (a good thing) and still allows you to analyze whether accuracy was systematically better or worse during certain parts of the year.

And as you write, the last two years have seen a big impact from COVID. I would say that any conclusions of the kind "method A outperformed method B in the test set, therefore we will use A henceforth" will be far more driven by COVID idiosyncrasies than by optimal or suboptimal train/test splits.

$\endgroup$
5
$\begingroup$
  • You want to have as many data points in the test set as possible.

  • You also want to have as many data points in the training set as possible.

Given a constrain on the total amount of data points, increasing the data points in the test set will reduce the number of data points in the training set.

So you will want to look for an optimum.


Example/Illustration

Below is an example where we change training set size and test set size. We do this in order to see if there is some optimum for the sizes that is balancing the pros and cons.

  • In the training we fit different models.
  • In the testing we select the model which is best.
  • We validate by computing the difference with the known theoretic function. (the theoretic function that we created ourselves, but we do the training and testing while pretending that we do not know the real theoretic function)

Let's use a model where we generate randomly 20 independent variables $X_i; 1\leq i \leq 20$ that are potentially in a linear model, which is in reality only made up from only the first five parameters $Y = X_1 + X_2 + X_3 + X_4 + X_5 + \text{noise}$. We do the fitting by regularization. Let's use 400 measurements for an experiment (which need to be divided into training and testing).

We repeat this experiment 100 times for each test set size.

Let's see how the regularization parameter varies based on the size of the test set.

enter image description here

Let's see how the performance parameter varies based on the size of the test set. (we plot the average of the 100 experiments per test size in red)

enter image description here

Interpretation of the graphs:

  • In the first graph we see: when we increase the test size then this increases the lambda parameter. We can see this as an increase in the control of 'overfitting' (which is the reason that we use a test data set).

  • In the second graph we see that there is some optimum for the test size. Increasing the test size initially improves the result, but at some point it does come with some costs. The reason is that increasing the test data size, decreases the training data size. When the training data size becomes smaller, then the models will perform less well.

In this particular example we see that this balance between large training data set (which improves the quality of the models) and the large test data set (which improves the control on overfitting) is optimal around a test data set of 33% of the entire data.


Note: The example indicates how the test data set size is influencing the results. But the result of 33% is not a general result. For each problem this size of the test data size differs. The optimum will depend on the amount of risks of overfitting versus risks of misfitting (noise).


R-code to produce graphs

library(glmnet)

set.seed(1)

n = 400
n_train = 20
vars = 20
lambda = 10^-4 * 1.5^c(1:60)

cvglmnet <- function(n_train) {
  
  ### create model matrix
  X <- matrix(rnorm(vars*n), ncol = vars)
  ### compute dependent variable 
  Y = rowSums(X[,1:5])+rnorm(n,0,10)
  
  # split data
  X_train = X[1:n_train,]
  Y_train = Y[1:n_train]
  X_test = X[(n_train+1):n,]
  Y_test = Y[(n_train+1):n]
  
  # perform model
  mod <- glmnet(X_train, Y_train, family = "gaussian", alpha = 0, lambda = lambda)
  
  ### compute mean train RSS
  Y_pred <- X_train %*% mod$beta
  errors <- Y_pred - Y_train
  RSS_train <- colSums(errors^2)/(n_train)
  
  ### compute mean test RSS
  Y_pred <- X_test %*% mod$beta
  errors <- Y_pred - Y_test
  RSS_test <- colSums(errors^2)/(n-n_train)
  
  ### the optimal model in the test 
  nm <- which.min(RSS_test)
  
  ### verify the model performance
  ### as distance in coefficients
  perf <- sum((mod$beta[,nm]-c(rep(1,5),rep(0,15)))^2)
  
  #plot(mod$beta[,nm])
  
  ### plotting
  #Ytot <- cbind(RSS_train,RSS_test)
  #plot(mod$lambda,RSS_train, log = "x", ylim = c(min(Ytot),max(Ytot)),
  #     ylab = "RSS", xlab = "lambda")
  #lines(mod$lambda,RSS_test)
  #points(mod$lambda[nm], RSS_test[nm], pch = 21, col = 1, bg = 2)
  
  return(list(lambda = mod$lambda[nm], performance = perf))
}

set.seed(1)
n_rep = 100
n_train <- seq(10,400,10)


### repeated computation of fitting procedure with different training/test data size
x <- replicate(n_rep,cvglmnet(n_train = n_train[1]))
Mlambda <- as.numeric(x[1,])
Mperformance <- as.numeric(x[2,])

for (i in 2:length(n_train)) {
  x <- replicate(n_rep,cvglmnet(n_train = n_train[i]))
  Mlambda <- cbind(Mlambda,as.numeric(x[1,])) 
  Mperformance <- cbind(Mperformance,as.numeric(x[2,])) 
}

plot(-1,-1, xlim = c(0,1), ylim = 10^c(-2,5), log = "y",
     xlab = "percent test size", ylab = "fitted lambda")

#plot(-1,-1, xlim = c(0,1), ylim = range(lambda), log = "y",
#     xlab = "percent test size", ylab = "fitted lambda")


for (i in 1:length(n_train)) {
  h <- Mlambda[,i]
  points(rep((n-n_train[i])/n,length(h)) + rnorm(length(h),0,0.01),h, pch = 21, 
         col = rgb(0,0,0,0.25), 
         bg = rgb(0,0,0,0.25), 
         cex = 0.7)
}



plot(-1,-1, xlim = c(0,1), ylim = c(0,15),
     xlab = "percent test size", ylab = "performance")

##plot(-1,-1, xlim = c(0,1), ylim = range(lambda), log = "y",
##     xlab = "percent test size", ylab = "fitted lambda")


for (i in 1:length(n_train)) {
  h2 <- Mperformance[,i]
  points(rep((n-n_train[i])/n,length(h2)) + rnorm(length(h2),0,0.01),h2, pch = 21, 
         col = rgb(0,0,0,0.25), 
         bg = rgb(0,0,0,0.25), 
         cex = 0.7)
  points((n-n_train[i])/n,mean(h2), pch = 21, col = 2, bg = 2)
}

 
$\endgroup$
2
  • $\begingroup$ +1: A really insightful empirical study $\endgroup$ Sep 22 at 16:33
  • $\begingroup$ Nice, +1! The question just is how this optimum split varies with the DGP. In a time series context, I could imagine it is strongly driven by the strength of seasonality and/or trend. $\endgroup$ Sep 23 at 7:03
3
$\begingroup$

I agree with the other answers, there are a lot of things that could go into an 'optimal' train test split for time series such as ensuring that you have complete cycles of seasonality. For example, if you have monthly data and only 28 months you are probably better off doing only a 4 month test split for models that require 2 full seasonal cycles and doing seasonal tests before hand. If it isn't seasonal then you could do longer test splits.

Similarly, if there are massive changes (such as COVID in this example) you will want the train split to have some of that data or else the best model will just be the most lucky one.

The most stable procedure in my experience is to select a test split that allows the models to see all the information they will need and then do time series cross validation through the test split.

I have also done some work with reinforcement learning where we feed the agent relevant time series features and allow it to choose the train-test split size to minimize actualized forecast error. It performed better than any static test size but worst than cross validation. So that could potentially work in general but probably not helpful for you right now. I would do a combination of both other answers with the last year held out and do time series cross validation to that split.

$\endgroup$
1
  • $\begingroup$ Enough to detect the patterns in the data is probably the correct answer. I use, for monthly data 50 data points because someone put forward a rule of thumb to that effect. But the answer depends on many things I am sure including the seasonality of your data. Structural breaks, where the pattern changes will make this a lot worse. $\endgroup$
    – user54285
    Sep 22 at 17:03
2
$\begingroup$

There is no theoretical optimum for this decision and will be highly problem-dependent.

One option to circumvent this decision is to use an online retraining model. At each point in time, train on previously observed data and test on the next point.

This is illustrated here under the section 'Walk Forward Validation'.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.