0
$\begingroup$

I need to do a naive time series forecasting as a benchmark. Therefore, I should split my dataset into train, validation, and test sets. If I divide the the dataset into train and test sets, the code will be as follows:

library(fpp2)
library(forecast)
data(ausbeer)
train <- window(ausbeer, start = c(1956,1), end = c(2007,4))
test <- window(ausbeer, start = c(2008,1))
naiveS <- snaive(train, h=1)
accuracy(naiveS, test)[,1:5]

How about if I want to use the data during 2005-2007 as my validation set? Could you please let me know how to compute the model's performance on the validation set? I read this post about sliding/rolling window and expanding window and this page. As I know the tscv is based on expanding window.

e <- tsCV(ausbeer, snaive, h=1)
sqrt(mean(e^2, na.rm=TRUE))

When we use a naive forecasting method, how to obtain the performance on the train, validation, and test sets?

Improve this question
$\endgroup$
4
  • $\begingroup$ It depends on what you mean by "validation" (and therefore also, "test") set - different people use the terms in different ways. If by "validation set" you mean "data for the final evaluation", then just adjust your windows and use what you call test. If you mean "intermediate evaluation set for tuning hyperparameters", then there is no such set, because the naive forecast does not have any hyperparameters to be tuned. $\endgroup$
    – Stephan Kolassa
    Feb 4 at 6:17
  • $\begingroup$ @StephanKolassa Thanks. I mean "intermediate evaluation set for tuning hyperparameters and selecting models". I need a naive forecast as benchmark to compare this naive model and other different models to each other, so I should obtain the performance of naive model on the validation set. $\endgroup$
    – ebrahimi
    Feb 4 at 13:35
  • $\begingroup$ If the validation set is an intermediate set that you only use to tune hyperparameters, then I don't see why you need the performance of the naive forecast on this set, because the naive forecast does not have any hyperparameters to tune. And on the other hand, if you do want the performance of the naive forecast on this set, you can simply set the window accordingly. I think I'm not completely understanding your question. $\endgroup$
    – Stephan Kolassa
    Feb 4 at 20:06
  • $\begingroup$ @StephanKolassa thanks. I want the performance of the naive forecast on this set, but I don't know how to set the window accordingly? $\endgroup$
    – ebrahimi
    Feb 4 at 21:21

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy