9
$\begingroup$

We are using STL (R implementation) for forecasting time series data.

Every day we run daily forecasts. We would like to compare forecast values with real values and identify average deviation. For example, we ran forecast for tomorrow and got forecast points, we would like to compare these forecasts points with real data we will obtain tomorrow. I am aware that forecasts values and real data may not match most of the times, that is one of the reason we would like to keep track of how much accurate we are each day.

Now we are trying identify what is the best approach to solve this problem? any help pointers would be appreciated.

I looked at Measuring forecast accuracy question, but it seems it is related to comparing models rather than calculating accuracy with real values.

I looked at accuracy function implementation in R, but confused with two questions:

1) Will it work on real data vs forecast data, because most of the tutorial saying as "test data" vs "forecast data"

2) It seems out put of accuracy function is array of values rather than % of deviation.

$\endgroup$
  • 1
    $\begingroup$ In order to get a great answer you must ask the right question - and that can be frustrating. Everyone wants to rule the world and that takes foreknowledge. If I had perfect knowledge of tomorrows stocks but had it today then I could make a ton of money/power/opportunity/glory/etc. If I were looking at your problem then I would want to see a histogram (or eCDF) of the predictive error. I might like to "jitter" the inputs of the forecast and look at its mean and variation, and compare the error to those. You must understand your error to fix it. $\endgroup$ – EngrStudent - Reinstate Monica Sep 10 '14 at 22:41
  • 1
    $\begingroup$ For the uninitiated, what is STL? $\endgroup$ – shadowtalker Sep 10 '14 at 23:50
  • $\begingroup$ @EngrStudent: "You must understand your error to fix it"- In this statement itself we have two parts. I am trying to find out possible approaches for first section itself. The methodology I pick will help me in picking strategies for part 2. $\endgroup$ – kosa Sep 11 '14 at 13:56
  • 1
    $\begingroup$ Mean is not the same thing as how a system behaves. Standard deviation is not the same thing as how a system behaves.Those two measures are summary statistics for measured system behavior. Error is not accuracy. Error is not uncertainty. Those two measures are summary statistics for error analogous to mean and standard deviation. Just like there are many infinities of measures of system behavior there are many infinities of measures of error behavior. What is your rubric? how to you measure a good way to think about error? $\endgroup$ – EngrStudent - Reinstate Monica Sep 11 '14 at 17:52
  • 1
    $\begingroup$ @Nambari - welcome to the world of the "wise". The beginning of knowledge is to know that you know nothing - to be a student. I try to be always learning myself, and try to be correctable by anyone speaking truth. If you play with the Eureqa tool, and try on appropriate sample data both every general form of "target expression" and every "error metric" then you will start to know this deep deep thing. I don't have a good answer. L'Hospital (aka L'Hopital) formulated the first least squared expression in 1696. A good start is the use case - where the mind comes in. What is that? $\endgroup$ – EngrStudent - Reinstate Monica Sep 11 '14 at 20:53
13
+100
$\begingroup$

There are many different ways of measuring forecast accuracy, and the accuracy() function from the forecast package for R outputs several of them. From your comment about "% of deviation" it sounds like you want to use Mean Absolute Percentage Error, which is one of the measures provided by accuracy(). The most common measures of forecast accuracy are discussed here. You might like to think about whether MAPE is the most appropriate measure for your problem, or whether one of the other measures is better.

The accuracy() function does work on real data. The "test data" are those data that were not used to construct the forecasts. Sometimes they are available but not used when the forecasts are computed (the classical split of data into training and test sets). In other situations, all the available data are used to compute the forecasts, and then you have to wait until there are some future observations available to use as the test data.

So if f is a vector of forecasts and x is a vector of observations corresponding to the same times, then

accuracy(f,x)

will do what you want.

$\endgroup$
  • $\begingroup$ "Measures based on percentage errors have the disadvantage of being infinite or undefined if yi=0 for any i in the period of interest, and having extreme values when any yi is close to zero." I think this will be issue in my case, because lot of cases actuals could be ZERO. I am thinking calculate MAE and change the results number to "percentage". Does it make sense? $\endgroup$ – kosa Sep 11 '14 at 14:02
  • $\begingroup$ Somehow my thank you note gone, really thank you for your time Dr Hyndman! $\endgroup$ – kosa Sep 11 '14 at 14:13
2
$\begingroup$

First, let's clarify that there are concepts of accuracy and precision. Accuracy is usually associated with a bias, i.e. systematic deviation of the forecast from the actuals. Precision is usually associated with the variance of the forecast errors. Something like this: $Accuracy=E(f)-y$ vs. $Precision=Var[f-y]$. So, when you mention "accuracy" in your post, were aware of the distinction?

Second, there are integrated measures of forecast quality, such as $MSFE=\frac{1}{n}\sum_{i=1}^n(f_i-y_i)^2$, where $f_i$ and $y_i$ are forecasts and actuals. There are statistics for this measure, such as Chow test for parameter constancy.

$\endgroup$
  • $\begingroup$ Thank you for answer! Yes, I am not worrying about precision at this moment. Just want to know the accuracy, "deviation of the forecast from actuals". I am not worrying about running few models calculate forecast errors and pick best model. My only aim is find out the deviation between actual and forecasted values. Our model is constant here. Irrespective of our model is good or bad for the data set, we just need the number of deviation. This question is not related to parameter fine tuning (or) model selection. I hope now I am clear. Please let me know if anything missing. $\endgroup$ – kosa Sep 11 '14 at 14:43
  • $\begingroup$ @Nambari, if you need the "number of deviations", why don't you use the number of deviations? Make a loop over the predictions, compare them with the real values and count the number of cases in which predictions differ from the real values. $\endgroup$ – Roman Sep 17 '14 at 10:32
2
$\begingroup$

I have been doing this in R here is my code for my data for both in-sample and out-of-sample data:

#accuracy testing for out-of-sample sample#

M<-#data#
deltaT<-#set observations per year,1/4 for quarterly, 1/12 for monthly
horiz<-#set amount of forecasts required
startY<-c(#,#) #set start date
N<-head(M,-horiz)
Nu<-log(Nu)
Nu<-ts(Nu,deltat=deltaT,start=startY)

#Run your forecasting method#
##My forecasting method is arima##

N<-#data#
N<-ts(N,deltat=deltaT,start=startY)
N<-tail(N,horiz)
fitted<-ts(append(fitted(Arimab), fArimab$mean[1]), deltat=deltaT, start = startY) #where Arimab is the ARIMA model and fArimab<-forecast(Arimab, h=horiz*2, simulate= TRUE, fan=TRUE)
N<-log(N)
fitted<-head(fitted,length(N))
error<-N-fitted
percenterror<-100*error/N
plus<-N+fitted
rmse<-function(error)
  sqrt(mean(error^2))
mae<-function(error)
  mean(abs(error))
mape<-function(percenterror)
  mean(abs(percenterror))
smape<-function(error,plus)
  mean(200*abs(error)/(plus))
mse<-function(error)
  mean(error^2)
me<-function(error)
  mean(error)
mpe<-function(percenterror)
  mean(percenterror)
accuracy<-matrix(c("rmse","mae","mape","smape","mse","me","mpe",(round(rmse(error),digits=3)),(round(mae(error),digits=3)),(round(mape(percenterror),digits=3)),(round(smape(error,plus),digits=3)),(round(mse(error),digits=3)),(round(me(error),digits=3)),(round(mpe(percenterror),digits=3))),ncol=2,byrow=FALSE)
View(accuracy,title="Accuracy of ARIMA out sample")

#Accuracy testing for the in sample

M<-#data#
deltaT<-#set observations per year,1/4 for quarterly, 1/12 for monthly
horiz<-#set amount of forecasts required
startY<-c(#,#) #set start date
Nu<-log(Nu)
Nu<-ts(Nu,deltat=deltaT,start=startY)
#run your forecasting method#
fitted<-ts(append(fitted(Arimab), fArimab$mean[1]), deltat=deltaT, start = startY)
N<-exp(Nu)
fitted<-exp(fitted)
fitted<-head(fitted,length(N))
error<-N-fitted
percenterror<-100*error/N
plus<-N+fitted
rmse<-function(error)
  sqrt(mean(error^2))
mae<-function(error)
  mean(abs(error))
mape<-function(percenterror)
  mean(abs(percenterror))
smape<-function(error,plus)
  mean(200*abs(error)/(plus))
mse<-function(error)
  mean(error^2)
me<-function(error)
  mean(error)
mpe<-function(percenterror)
  mean(percenterror)
accuracy<-matrix(c("rmse","mae","mape","smape","mse","me","mpe",(round(rmse(error),digits=3)),(round(mae(error),digits=3)),(round(mape(percenterror),digits=3)),(round(smape(error,plus),digits=3)),(round(mse(error),digits=3)),(round(me(error),digits=3)),(round(mpe(percenterror),digits=3))),ncol=2,byrow=FALSE)
View(accuracy,title="Accuracy of ARIMA in sample")

hope this helps a bit. if you want my full code i used to run this please ask as this is very basic

$\endgroup$
1
$\begingroup$

The short answer: to evaluate the quality of your predictions, use exactly the same measure that you used in the training (fitting) of your model.

The long answer:

In order to chose a measure for the accuracy of your forecasts, your first need to know how you interpret you predictions. In other words, what do you actually give as a "forecast"? Is it mean value? Median? Most probable value? The answer on this question will uniquely identify the measure of the forecast accuracy. If you predict mean, you have to use the root mean square deviation as the measure of the forecast accuracy. If you predict median you have to use mean absolute deviation as the measure of accuracy.

I will elaborate a bit on this point. Let us assume that you make a prediction / forecast for tomorrow. Let us also assume that for any value that you might observe tomorrow you have a corresponding probability to be observed. For example you know that you might observe 1 with probability 0.03, 2 with probability 0.07, 3 with probability 0.11, and so on. So, you have a distribution of probabilities over different values. Having this distribution you can calculate different properties and give them as your "predictions". You can calculate mean and give it as the prediction for tomorrow. Alternatively you can use median as your prediction. You can also find the most probable value and give it as your prediction for tomorrow.

If you use mean value as prediction, than the question of "how to measure the accuracy of my prediction" has to be replaced by "what is the measure of the accuracy for the mean" and the answer is "root mean square deviation between the real values and prediction". If you use median as predictions, you have to use mean absolute deviation.

It might be that you do not know if you use median or mean or something else. To find out what you actually use as predictions you have to know what measure you try to minimize in the training. If you try to find parameters of the model that minimize root mean square deviation between the predictions and target values from the training data, then your predictions have to be treated as mean. If you minimize absolute deviations, then you train your model to provide medians and so on.

ADDED

I would like to emphasize one thing. As I have mentioned above it is important to keep the same measure of accuracy in "fit" and in "predict". In addition to that I would like to say that you are absolutely free in choosing your measures. There are no "better" or "worse" measures. The measure should be determined by the way you (or your client) use your predictions. For example it might be very important (to you or your client) to have an exact match and if you do not have it, it does not play any role if the difference between the real and predicted values is big or small. In other cases this difference plays a role. Difference of 1 is better than difference of 2. In some cases difference of 2 is 2 time worse than difference of 1. In other cases difference equal to 2 is 100 times worse than difference equal to 1. You can also imagine exotic cases in which you need to generate a value that differs from observations. So, measure of quality of the numbers that you generate can be whatever you want, depending on what you need. What is important, is to use the same measure in training (fit) and evaluation of predictions.

$\endgroup$
  • $\begingroup$ (Related to your comment on other answer)In most of the cases predictions differ from real values, I don't think in any case we can get perfect fit. So, the approach you suggested may not be ideal, because we will get 100%. But what I am thinking is get the difference between actual vs prediction in percentage, which is nothing but MAPE. The case we are handling has very high chances of getting ZERO actual very frequently, due to circumstances, in which case MAPE may not be best option because percentage will be INFINITY. This is where I was stuck. $\endgroup$ – kosa Sep 17 '14 at 14:42
  • $\begingroup$ I know MAPE is the one in principle I want, but my data set has this unfortunate case where REAL values in series can be ZERO very frequently. $\endgroup$ – kosa Sep 17 '14 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.