We come to this toy example showing MAPE and MASE are not consistent when measuring forecasting accuracy.
Data consist of 100 white noise and 100 $AR(1)$ time series with length $N=500$, mean $\mu=1$ and standard deviation $\sigma=1$.
# parameters
N <- 500
mu <- 10
sigma <- 1
# generate white noise
set.seed(1)
WNts <- list(NULL)
for (i in 1:100){
WNts[[i]] <- ts(rnorm(N,mu,sigma))}
# generate AR(1)
ar1 <- list(NULL)
for (i in 1:100){
ar1[[i]] <- arima.sim(model=list(ar=c(0.7)),n=N,sd=sqrt(sigma-0.7^2))+mu}
# data used
SimData <- c(WNts,ar1)
Each time series are split into training and test set. What we are looking at are the MAPE and MASE on test set. To take a further look at MASE, we also calculate MAE and Q, which are numerator and denominator of MASE.
# forecasting accuracy on test set
SimDataAccuracy <- foreach (i = 1:200,.combine = rbind)%dopar%{
x <- SimData[[i]]
trainx <- window(x,end=400)
testx <- window(x,start=401,end=500)
fit <- auto.arima(trainx)
accuracyArima <- accuracy(forecast(fit,100),testx)
Q <- mean(abs(diff(trainx)))
c(accuracyArima[2,5],accuracyArima[2,6],accuracyArima[2,3],Q)
}
colnames(SimDataAccuracy) <- c("MAPE","MASE","MAE","Q")
# plot
par(mfrow=c(2,2))
# MAPE
plot(SimDataAccuracy[,1],ylab='MAPE',xlab='')
# MASE
plot(SimDataAccuracy[,2],ylab='MASE',xlab='')
# MAE
plot(SimDataAccuracy[,3],ylab='MAE (numerator of MASE)',xlab='')
# Q
plot(SimDataAccuracy[,4],ylab='Q (denomintor of MASE)',xlab='')
The plots show forecasting on white noise has smaller MASE just because of the larger Q. From both MAPE and MAE, white noise and $AR(1)$ time series have rather similar forecasting accuracy.
Does that mean
White noise is easier to predict? (I cannot see a reason), or
They have similar forecastability and MASE is telling some disturbing information here?