Forecast accuracy calculation

Question

We are using STL (R implementation) for forecasting time series data.

Every day we run daily forecasts. We would like to compare forecast values with real values and identify average deviation. For example, we ran forecast for tomorrow and got forecast points, we would like to compare these forecasts points with real data we will obtain tomorrow. I am aware that forecasts values and real data may not match most of the times, that is one of the reason we would like to keep track of how much accurate we are each day.

Now we are trying identify what is the best approach to solve this problem? any help pointers would be appreciated.

I looked at Measuring forecast accuracy question, but it seems it is related to comparing models rather than calculating accuracy with real values.

I looked at accuracy function implementation in R, but confused with two questions:

1) Will it work on real data vs forecast data, because most of the tutorial saying as "test data" vs "forecast data"

2) It seems out put of accuracy function is array of values rather than % of deviation.

Answer 1

There are many different ways of measuring forecast accuracy, and the accuracy() function from the forecast package for R outputs several of them. From your comment about "% of deviation" it sounds like you want to use Mean Absolute Percentage Error, which is one of the measures provided by accuracy(). The most common measures of forecast accuracy are discussed here. You might like to think about whether MAPE is the most appropriate measure for your problem, or whether one of the other measures is better.

The accuracy() function does work on real data. The "test data" are those data that were not used to construct the forecasts. Sometimes they are available but not used when the forecasts are computed (the classical split of data into training and test sets). In other situations, all the available data are used to compute the forecasts, and then you have to wait until there are some future observations available to use as the test data.

So if f is a vector of forecasts and x is a vector of observations corresponding to the same times, then

accuracy(f,x)

will do what you want.

Answer 2

First, let's clarify that there are concepts of accuracy and precision. Accuracy is usually associated with a bias, i.e. systematic deviation of the forecast from the actuals. Precision is usually associated with the variance of the forecast errors. Something like this: $Accuracy=E(f)-y$ vs. $Precision=Var[f-y]$. So, when you mention "accuracy" in your post, were aware of the distinction?

Second, there are integrated measures of forecast quality, such as $MSFE=\frac{1}{n}\sum_{i=1}^n(f_i-y_i)^2$, where $f_i$ and $y_i$ are forecasts and actuals. There are statistics for this measure, such as Chow test for parameter constancy.

Answer 3

I have been doing this in R here is my code for my data for both in-sample and out-of-sample data:

#accuracy testing for out-of-sample sample#

M<-#data#
deltaT<-#set observations per year,1/4 for quarterly, 1/12 for monthly
horiz<-#set amount of forecasts required
startY<-c(#,#) #set start date
N<-head(M,-horiz)
Nu<-log(Nu)
Nu<-ts(Nu,deltat=deltaT,start=startY)

#Run your forecasting method#
##My forecasting method is arima##

N<-#data#
N<-ts(N,deltat=deltaT,start=startY)
N<-tail(N,horiz)
fitted<-ts(append(fitted(Arimab), fArimab$mean[1]), deltat=deltaT, start = startY) #where Arimab is the ARIMA model and fArimab<-forecast(Arimab, h=horiz*2, simulate= TRUE, fan=TRUE)
N<-log(N)
fitted<-head(fitted,length(N))
error<-N-fitted
percenterror<-100*error/N
plus<-N+fitted
rmse<-function(error)
  sqrt(mean(error^2))
mae<-function(error)
  mean(abs(error))
mape<-function(percenterror)
  mean(abs(percenterror))
smape<-function(error,plus)
  mean(200*abs(error)/(plus))
mse<-function(error)
  mean(error^2)
me<-function(error)
  mean(error)
mpe<-function(percenterror)
  mean(percenterror)
accuracy<-matrix(c("rmse","mae","mape","smape","mse","me","mpe",(round(rmse(error),digits=3)),(round(mae(error),digits=3)),(round(mape(percenterror),digits=3)),(round(smape(error,plus),digits=3)),(round(mse(error),digits=3)),(round(me(error),digits=3)),(round(mpe(percenterror),digits=3))),ncol=2,byrow=FALSE)
View(accuracy,title="Accuracy of ARIMA out sample")

#Accuracy testing for the in sample

M<-#data#
deltaT<-#set observations per year,1/4 for quarterly, 1/12 for monthly
horiz<-#set amount of forecasts required
startY<-c(#,#) #set start date
Nu<-log(Nu)
Nu<-ts(Nu,deltat=deltaT,start=startY)
#run your forecasting method#
fitted<-ts(append(fitted(Arimab), fArimab$mean[1]), deltat=deltaT, start = startY)
N<-exp(Nu)
fitted<-exp(fitted)
fitted<-head(fitted,length(N))
error<-N-fitted
percenterror<-100*error/N
plus<-N+fitted
rmse<-function(error)
  sqrt(mean(error^2))
mae<-function(error)
  mean(abs(error))
mape<-function(percenterror)
  mean(abs(percenterror))
smape<-function(error,plus)
  mean(200*abs(error)/(plus))
mse<-function(error)
  mean(error^2)
me<-function(error)
  mean(error)
mpe<-function(percenterror)
  mean(percenterror)
accuracy<-matrix(c("rmse","mae","mape","smape","mse","me","mpe",(round(rmse(error),digits=3)),(round(mae(error),digits=3)),(round(mape(percenterror),digits=3)),(round(smape(error,plus),digits=3)),(round(mse(error),digits=3)),(round(me(error),digits=3)),(round(mpe(percenterror),digits=3))),ncol=2,byrow=FALSE)
View(accuracy,title="Accuracy of ARIMA in sample")

hope this helps a bit. if you want my full code i used to run this please ask as this is very basic

Answer 4

The short answer: to evaluate the quality of your predictions, use exactly the same measure that you used in the training (fitting) of your model.

The long answer:

In order to chose a measure for the accuracy of your forecasts, your first need to know how you interpret you predictions. In other words, what do you actually give as a "forecast"? Is it mean value? Median? Most probable value? The answer on this question will uniquely identify the measure of the forecast accuracy. If you predict mean, you have to use the root mean square deviation as the measure of the forecast accuracy. If you predict median you have to use mean absolute deviation as the measure of accuracy.

I will elaborate a bit on this point. Let us assume that you make a prediction / forecast for tomorrow. Let us also assume that for any value that you might observe tomorrow you have a corresponding probability to be observed. For example you know that you might observe 1 with probability 0.03, 2 with probability 0.07, 3 with probability 0.11, and so on. So, you have a distribution of probabilities over different values. Having this distribution you can calculate different properties and give them as your "predictions". You can calculate mean and give it as the prediction for tomorrow. Alternatively you can use median as your prediction. You can also find the most probable value and give it as your prediction for tomorrow.

If you use mean value as prediction, than the question of "how to measure the accuracy of my prediction" has to be replaced by "what is the measure of the accuracy for the mean" and the answer is "root mean square deviation between the real values and prediction". If you use median as predictions, you have to use mean absolute deviation.

It might be that you do not know if you use median or mean or something else. To find out what you actually use as predictions you have to know what measure you try to minimize in the training. If you try to find parameters of the model that minimize root mean square deviation between the predictions and target values from the training data, then your predictions have to be treated as mean. If you minimize absolute deviations, then you train your model to provide medians and so on.

ADDED

I would like to emphasize one thing. As I have mentioned above it is important to keep the same measure of accuracy in "fit" and in "predict". In addition to that I would like to say that you are absolutely free in choosing your measures. There are no "better" or "worse" measures. The measure should be determined by the way you (or your client) use your predictions. For example it might be very important (to you or your client) to have an exact match and if you do not have it, it does not play any role if the difference between the real and predicted values is big or small. In other cases this difference plays a role. Difference of 1 is better than difference of 2. In some cases difference of 2 is 2 time worse than difference of 1. In other cases difference equal to 2 is 100 times worse than difference equal to 1. You can also imagine exotic cases in which you need to generate a value that differs from observations. So, measure of quality of the numbers that you generate can be whatever you want, depending on what you need. What is important, is to use the same measure in training (fit) and evaluation of predictions.