1
$\begingroup$

I used two different methods to forecast a time series data.

  1. The first one used is HoltWinters with Beta and Gamma as FALSE, since I don't see any trend or seasonality in the plot.

Below is the result from Box.test

Box.test(fore.holt.stat$residuals, type="Ljung-Box", lag=10)

Box-Ljung test
data:  fore.holt.stat$residuals
X-squared = 10.691, df = 10, p-value = 0.3821

The p-value is 0.3821

  1. I used auto.arima on the data and below is the result

    Box.test(fore.arima$residuals, type="Ljung-Box", lag=10)

Box-Ljung test
data:  fore.arima$residuals
X-squared = 14.724, df = 10, p-value = 0.1425

The p-value is 0.14

Question 1 :

Can I say that the first model is better since I have a higher p-vale?

Below are few other observations :

Model 1:

accuracy(fore.holt.stat)
               ME     RMSE      MAE       MPE     MAPE     MASE       ACF1
Training set 424.9864 10275.55 7930.602 0.8782302 9.251837 0.766108 0.02142331

Model 2:

accuracy(fore.arima)
               ME     RMSE     MAE        MPE     MAPE      MASE      ACF1
Training set 284.5242 7243.413 5371.42 -0.1874984 6.036736 0.5183941 0.0100049


Question 2 :

Which of the model is correct based on the accuracy function output?

In both the models, the p-value is high, but the mean of errors is not close to zero.

Improve this question
$\endgroup$
1
  • $\begingroup$ Google 'forecast evaluation". It's a big area of forecasting. There are many criteria. $\endgroup$
    – Aksakal
    Commented Sep 28, 2015 at 18:34

1 Answer 1

Reset to default
1
$\begingroup$

Can I say that the first model is better since I have a higher p-vale?

It depends on how you define what is "better". But basically you are right because lower autocorrelation of residuals is more desirable than higher autocorrelation. Also, check out this post on the use of Ljung-Box test versus Breusch-Godfrey test.

Which of the model is correct based on the accuracy function output?

None of the models needs to be correct. Normally it is not the case; models are simplifications of reality. However, one of the models may be more useful than the other. It looks as if the second model has better forecast accuracy based on RMSE, MAE, MAPE, and MASE. (I did not check what the other acronyms stand for, so I will not comment on those.) MASE below unity in both cases looks encouraging, although context is needed to make more precise conclusions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Taking into account just the accuracy of the forecast : 1. Model 1 has higher p-value and Model 2 has lower values for MASE etc. Which out of the two can is say is a better prediction? 2. For any forecast, is it not like ME(mean of errors) should be close to 0. Here I am seeing 424 and 284. $\endgroup$
    – Ashish Anand
    Commented Sep 29, 2015 at 4:55
  • 1
    $\begingroup$ $p$-value has nothing to do with forecast accuracy, at least directly. Thus when interested in forecast accuracy, just look at the output of the accuracy function. That points to model 2 being preferred to model 1. However, I share your concern about the mean error. Ideally, ME indeed should be close to zero. If your observations are not measured in billions or so, the current ME looks poor. Maybe you are trying to fit the model to a data that clearly cannot be described by this kind of model?.. $\endgroup$
    – Richard Hardy
    Commented Sep 30, 2015 at 6:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.