37
$\begingroup$

I recently applied a range of forecasting methods (MEAN, RWF, ETS, ARIMA and MLPs) and found that MEAN did surprisingly well. (MEAN: where all future predictions are predicted as been equal to the arithmetic mean of the observed values.) MEAN even outperformed ARIMA on the three series I used.

What I want to know is if this is unusual? Does this mean the times series I'm using are strange? Or does this indicate that I've set something up wrong?

$\endgroup$
  • $\begingroup$ en.wikipedia.org/wiki/Martingale_%28probability_theory%29 $\endgroup$ – Mehrdad Nov 21 '14 at 16:27
  • $\begingroup$ @Mehrdad one could definitely craft a nice answer around Martingales. $\endgroup$ – shadowtalker Nov 21 '14 at 19:18
  • 1
    $\begingroup$ It's quite common for simple methods to perform well, especially out-of-sample (which is where it matters). This effect becomes stronger on short series. If there's not much serial correlation or trend, then we'd expect the mean to do very well even with relatively long series. $\endgroup$ – Glen_b Nov 25 '14 at 9:07
31
$\begingroup$

I'm a practitioner, both producer and user of forecasting and NOT a trained statistician. Below I share some of my thoughts on why your mean forecast turned out better than ARIMA by referring to research article that rely on empirical evidence. One book that time and time again I go back to refer is the Principles of Forecasting book by Armstrong and its website which I would recommend as an excellent read for any forecaster, provides great insight on usage and guiding principles of extrapolation methods.

To answer you first question - What I want to know is if this is unusual?

There is a chapter called Extrapolation for Time-Series and Cross-Sectional Data which also available free in the same website. The following is the quote from the chapter

"For example, in the real-time M2-competition, which examined 29 monthly series, Box-Jenkins proved to be one of the least-accurate methods and its overall median error was 17% greater than that for a naive forecast"

There lies an empirical evidence on why your mean forecasts was better than ARIMA models.

There is also been study after study in empirical competitions and the third M3 competition that show Box - Jenkins ARIMA approach fails to produce accurate forecast and lacks evidence that it performs better for univariate trend extrapolation.

There is also another paper and an ongoing study by Greene and Armstrong entitled "Simple Forecasting: Avoid Tears Before Bedtime" in the same website. The authors of the paper summarize as follows:

In total we identified 29 papers incorporating 94 formal comparisons of the accuracy of forecasts from complex methods with those from simple—but not in all cases sophisticatedly simple—methods. Eighty-three percent of the comparisons found that forecasts from simple methods were more accurate than, or similarly accurate to, those from complex methods. On average, the errors of forecasts from complex methods were about 32 percent greater than the errors of forecasts from simple methods in the 21 studies that provide comparisons of errors

To answer your third question: does this indicate that I've set something up wrong? No, I would aconsider ARIMA as complex method and Mean forecast as simple methods. There is ample evidence that simple methods like Mean forecast outperform complex methods like ARIMA.

To answer your second question: Does this mean the times series I'm using are strange?

Below are what I considered to be experts in real world forecasting:

  • Makridakis (Pioneered Empirical competition on Forecasting called M, M2 and M3, and paved way for evidence based methods in forecasting)
  • Armstrong (Provides valuable insights in the form of books/articles on Forecasting Practice)
  • Gardner (Invented Damped Trend exponential smoothing another simple method which works surprisingly well vs. ARIMA)

All of the above researchers advocate, simplicity (methods like your mean forecast) vs. Complex methods like ARIMA. So you should feel comfortable that your forecasts are good and always favor simplicity over complexity based on empirical evidence. These researchers have all contributed immensely to the field of applied forecasting.

In addition to Stephan's good list of simple forecasting method. there is also another method called Theta forecasting method which is a very simple method (basically Simple Exponential smoothing with a drift that equal 1/2 the slope of linear regression) I would add this to your toolbox. Forecast package in R implements this method.

$\endgroup$
  • 1
    $\begingroup$ I really like the perspective you provide on forecasting and the evidence you adduce to support it, but elements of this answer are objectionable because they read too much like a rant against "statisticians" or perhaps against formal statistical training--and are flat out wrong. For instance, Makridakis' advanced degrees are in (guess what?) statistics, that's what he teaches, and that's what he does. $\endgroup$ – whuber Nov 21 '14 at 23:02
  • 1
    $\begingroup$ Got it, I'll remove it. I was trying to make a point that major advances on evidence based methods came from non statisticians. But I see your point that it might come across like that $\endgroup$ – forecaster Nov 21 '14 at 23:04
  • $\begingroup$ Done, Also, Makridakis PHD was in Management Information Systems according to this published interview $\endgroup$ – forecaster Nov 21 '14 at 23:18
  • $\begingroup$ FWIW, his LinkedIn page--which he maintains--lists both his PhDs in statistics. But the argument is pointless: claiming that somebody is not a statistician because their degree might not specifically be in statistics has little value and is beside the point here. (Until very recently most people whose career was in statistics had degrees in other fields because there were few statistics programs available.) $\endgroup$ – whuber Nov 22 '14 at 1:22
  • 2
    $\begingroup$ +1. However, you write "There lies an empirical evidence on why your mean forecasts was better than ARIMA models." - no, this only tells us that the mean was better (in this specific instance), not why. I'd turn the argument around and put the burden of proof on ARIMA and other models. I have never understood quite why the data-generating process should care about past errors against my model, which is what MA terms essentially model. My personal suspicion is that ARIMA is so popular because you can actually prove stuff, like unit roots and stationarity - not because it forecasts well. $\endgroup$ – Stephan Kolassa Nov 25 '14 at 19:19
35
$\begingroup$

This is not at all surprising. In forecasting, you very often find that extremely simple methods, like

  • the overall mean
  • the naive random walk (i.e., the last observation used as a forecast)
  • a seasonal random walk (i.e., the observation from one year back)
  • Single Exponential Smoothing

outperform more complex methods. That is why you should always test your methods against these very simple benchmarks.

A quote from George Athanosopoulos and Rob Hyndman (who are experts in the field):

Some forecasting methods are very simple and surprisingly effective.

Note how they explicitly say they will be using some very simple methods as benchmarks.

In fact, their entire free open online textbook on forecasting is very much recommended.

EDIT: One of the better-accepted forecast error measures, the Mean Absolute Scaled Error (MASE) by Hyndman & Koehler (see also here) measures how much a given forecast improves on the (in-sample) naive random walk forecast: if MASE < 1, your forecast is better than the in-sample random walk. You'd expect this to be an easily beaten bound, right?

Not so: sometimes, even the best out of multiple standard forecasting methods like ARIMA or ETS will only yield a MASE of 1.38, i.e., be worse (out-of-sample) than the (in-sample) random walk forecast. This is sufficiently disconcerting to generate questions here. (That question is not a duplicate of this one, since the MASE compares out-of-sample accuracy to in-sample accuracy of a naive method, but it is also enlightening for the present question.)

$\endgroup$
  • 1
    $\begingroup$ Thanks for accepting, but maybe you want to wait for a day - if a question has accepted answers, fewer people will even read it, let alone comment or answer. And other people may have different takes on this. Feel free to un-accept ;-) $\endgroup$ – Stephan Kolassa Nov 21 '14 at 14:05
  • $\begingroup$ Its very honest of you :) I'll give it a day. Thanks. $\endgroup$ – Andy T Nov 21 '14 at 14:06
  • 8
    $\begingroup$ "complex" is a close relative of "overfitted." $\endgroup$ – shadowtalker Nov 21 '14 at 19:18
  • 1
    $\begingroup$ +1 nice answer. If forecasting was an evidence based field like medicine, ARIMA method would be history. $\endgroup$ – forecaster Nov 21 '14 at 22:47
  • 2
    $\begingroup$ simple minded ARIMA without validating the Gaussian Assumptions is already history for most of us BUT apparently not for all ! $\endgroup$ – IrishStat Nov 21 '14 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.