1
$\begingroup$

the tsCV function from the forecast package allows for forecasting using a rolling forecast window. Why is it that the step-size is always 1 sample ahead, and the function doesn't allow for stepping k samples for each forecast?

My business scenario is that I have to predict 24 samples ahead (h=24), but I'm forecasting only at midnight of each day (k=24), it's not that I have to predict 24 hours ahead every hour. I would like to simulate that using tsCV.

Note that in the sklearn counterpart TimeSeriesSplit this functionality is implemented, and the difference between successive training sets isn't hard-coded to 1 sample.

$\endgroup$
1
$\begingroup$

If your method is in the forecast class and you have more than 24 observations, you should have no problem in running tsCV with h = 24. It will return a matrix with h = 24 columns and the number of rows equals to your data size.

If you only want the cv errors for K=24, then you can do the following: if d is the number of days in your data you, the code

tsCV(data, forecast_method, h = 24)[24*(1:(d-1)),]

should return a error matrix for the whole day predicting only midnight (K=24). The code below illustrates this (d=5)

set.seed(10)
X <- ts(rnorm(5*24))
tsCV(X, rwf, h = 24)[24*(1:4),]

It returns a $4\times 24$ matrix, each row contains the prediction at midnight for some hour of that day, which is given by the column. For instance, element [2,15] is the prediction error at midnight of the second day for hour 15. Then you can apply a metric you see fit to choose the forecast method.

Edit: As you have stated, this is a hack, and quite a slow one actually since it runs 24X more than you actually need. I went to the source code of tsCV and changed it to add a "step" parameter.

The code for the function definition goes below, just copy and run this in a R script. I commented on the lines that I added or modified.

tsCV_step <- function (y, forecastfunction, h = 1, window = NULL, xreg = NULL, 
          initial = 0, step = 1, ...) 
{
  y <- as.ts(y)
  n <- length(y)
  step <- round(step)
  step_ind <- seq(step, n - 1L, by = step) ### Added line

  e <- ts(matrix(NA_real_, nrow = floor(n/step), ncol = h)) ### Modified line: n was replaced by floor(n/step)
  if (initial >= n) 
    stop("initial period too long")
  tsp(e)[-2] <- tsp(y)[-2] ### Modified line: [-2] added
  if (!is.null(xreg)) {
    xreg <- ts(as.matrix(xreg))
    if (NROW(xreg) != length(y)) 
      stop("xreg must be of the same size as y")
    tsp(xreg) <- tsp(y)
  }
  if (is.null(window)) 
    indx <- seq(1 + initial, n - 1L)
  else indx <- seq(window + initial, n - 1L, by = 1L)
  indx <- intersect(indx, step_ind) ### Added line
  for (i in indx) {
    y_subset <- subset(y, start = ifelse(is.null(window), 
                                         1L, ifelse(i - window >= 0L, i - window + 1L, stop("small window"))), 
                       end = i)
    if (is.null(xreg)) {
      fc <- try(suppressWarnings(forecastfunction(y_subset, 
                                                  h = h, ...)), silent = TRUE)
    }
    else {
      xreg_subset <- as.matrix(subset(xreg, start = ifelse(is.null(window), 
                                                           1L, ifelse(i - window >= 0L, i - window + 1L, 
                                                                      stop("small window")))))
      fc <- try(suppressWarnings(forecastfunction(y_subset, 
                                                  h = h, xreg = xreg_subset, ...)), silent = TRUE)
    }
    if (!is.element("try-error", class(fc))) {
      e[i/step, ] <- y[i + (1:h)] - fc$mean ### Modified Line: i replaced by i/step 
                                            ### for "e" first index
    }
  }
  if (h == 1) {
    return(e[, 1L])
  }
  else {
    colnames(e) <- paste("h=", 1:h, sep = "")
    return(e)
  }
}

The following example illustrates that it gives the same result as before and is way faster: tsCV_step took approximately 0.4 seconds to compute, while tsCV took more than 10 seconds. The all(step_err == no_step_err, na.rm = T) output shows that the hack output and tsCV_step output is the same, with the only difference that the hack goes to d now instead of (d-1), so the NA's will appear.

set.seed(10)
days = 100
data <- ts(rnorm(24*days))
K_step = 24

# Compute the errors with step and the time taken
init <- Sys.time()
step_err <- tsCV_step(data, rwf, h = 24, step = K_step)
tot_time_step <- Sys.time() - init


init <- Sys.time()
no_step_err <- tsCV(data, rwf, h = 24)[K_step*(1:d),]
tot_time_no_step <- Sys.time() - init

cat(paste("tsCV_step time taken: ", tot_time_step, "\n",
          "tsCV time taken: ", tot_time_no_step, sep = ""))


all(step_err == no_step_err, na.rm = T)

Obs1: I'm no expert on R programming, so I might have introduced a bug by changing the source code. It seems stable for your application if you are not using the other parameters like window and xreg.

Obs2: The NA's are also reported, just like in tsCV.

$\endgroup$
  • 1
    $\begingroup$ I did something similar: ret <- ret[seq(attributes(ret)$dim[1]) %% 24 == 1,] - but this is a hack, as the tsCV runs X24 times than necessary, right? (it's very slow when using a complicated forecast...) $\endgroup$ – ihadanny Oct 15 at 14:02
  • $\begingroup$ Yes, I will try to think how to overcome that problem. I will edit my answer to clarify that! $\endgroup$ – Lucas Prates Oct 15 at 15:18
  • $\begingroup$ wow thanks!!! checking if there's a simpler solution before accepting $\endgroup$ – ihadanny Oct 15 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.