6
$\begingroup$

I have created training set and test set from my data. Then I performed auto.arima() and ets() in R on the training set to predict one-step ahead forecasts. These were then compared with the test set values to measure error, namely RMSE, MAPE & MAE.

This is the output of both ets and auto.arima

 RMSE.ets
 [1] 3767.561
 RMSE.ar
 [1] 3776.308
 MAE.ets
 [1] 2885.112
 MAE.ar
 [1] 2624.482
 MAPE.ets
 [1] 0.04232065
 MAPE.ar
 [1] 0.03857747

Which criteria should be ideally used to select one of the two models (ets or auto.arima) for future predictions. Or is there any other criteria that I am missing out on.

Kindly help.

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ What is 'best' depends on what you want to optimize. You may want to focus on the ones that relate to prediction error in this instance. $\endgroup$
    – Glen_b
    May 20, 2013 at 7:22
  • 1
    $\begingroup$ This paper argues that MAE is best for some model comparisons: Willmott, C. J. and K. Matsuura, 2005: Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30 (1), 79. $\endgroup$
    – Marc in the box
    May 20, 2013 at 13:17
  • $\begingroup$ Question for Priyaj: When you say accuracy do you mean (1) the difference between the mean error and the true value, (2) a measure of the width of variation of errors, or (3) the measure of the appropriateness of the particular/fitted model in representing the information and not the noise given the particular set of training data? $\endgroup$
    – EngrStudent
    May 20, 2013 at 15:07
  • 2
    $\begingroup$ As @Glen_b notes, this very much depends on what you want to do with your forecasts, i.e., your loss function. What kinds of errors hurt you the most? Will a slightly biased but accurate forecast be better than an unbiased but highly variable one? You will need to think about what subsequent decisions are driven by the forecast... $\endgroup$
    – Stephan Kolassa
    Apr 29, 2016 at 20:45
  • 2
    $\begingroup$ ... Right now, the question is somewhat similar to "I know horsepower, top speed and price, how can I decide which car is best for me?" You first need to decide whether you need to take kids to school or not, whether you need to drive only two miles back and forth every day or sixty, whether you are a plumber and need to carry all of your tools and spare parts around and so forth. $\endgroup$
    – Stephan Kolassa
    Apr 29, 2016 at 20:47

1 Answer 1

Reset to default
4
$\begingroup$

I have to agree with Glen.

It is axiomatic in control system's engineering that there is no such thing as "best" without a measure of goodness.

Some (weak) examples of candidate bests include:

  • Best = robust indicator of central tendency
  • Best = robust indicator of variation around central tendency
  • Best = fastest to compute

Personally, when trying to select models, I like to use AICc because it is "good enough". It accounts for over-fitting, has a fair basis in statistics, and is comprised using figures of merit that many systems have as outputs.

Here is some info on it: http://www4.ncsu.edu/~shu3/Presentation/AIC.pdf

One of its family members is BIC (Bayes Information Criterion): link1,link2. You might want to explore "Information Criterion" for model selection.

You might consider using "Akaike weights" to combine your models for better predictive power.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ First of all, it's not my recommendation; I have referred the questioner to a peer reviewed paper, and have not given an answer myself. Second, AIC gives a measure of significance, not the accuracy of prediction. $\endgroup$
    – Marc in the box
    May 20, 2013 at 14:40
  • $\begingroup$ Updated the answer. AIC can be computed using the sum of squared error (rss), the number of samples, and the number of parameters in the model. For models of equal parameter count, it reduces to a comparison of errors. This is a measure of precision but not of accuracy. In English there is ambiguity in the usage of those terms. I should ask for clarification there. $\endgroup$
    – EngrStudent
    May 20, 2013 at 15:02
  • 2
    $\begingroup$ While I agree with you in general, the AICc does not help here as the question involves comparing across model classes where the AICc is not comparable. $\endgroup$
    – Rob Hyndman
    May 20, 2013 at 23:15
  • 1
    $\begingroup$ -1. As @RobHyndman notes, AIC(c) cannot be compared between ets and auto.arima models, and the OP explicitly asks about exactly this comparison. Plus, AIC(c) is usually (and historically, from Akaike's original papers) only computed in-sample, not out-of-sample - it provides guidance about which model is closest to an unknown data-generating process, which is related to, but not identical to the question which model yields the best out-of-sample accuracy - and we still haven't gone into the question of what "best accuracy" means. $\endgroup$
    – Stephan Kolassa
    Apr 29, 2016 at 20:43
  • $\begingroup$ @StephanKolassa - if you noticed, I did not say it was the best for his case. I said "there is no best without a measure of goodness". I like to use information criterion in my work and recommended the asker explore it, but did not state its applicability to either ets or auto.arima. $\endgroup$
    – EngrStudent
    Apr 30, 2016 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.