I recently applied a range of forecasting methods (MEAN, RWF, ETS, ARIMA and MLPs) and found that MEAN did surprisingly well. (MEAN: where all future predictions are predicted as been equal to the arithmetic mean of the observed values.) MEAN even outperformed ARIMA on the three series I used.
What I want to know is if this is unusual? Does this mean the times series I'm using are strange? Or does this indicate that I've set something up wrong?
I'm a practitioner, both producer and user of forecasting and NOT a trained statistician. Below I share some of my thoughts on why your mean forecast turned out better than ARIMA by referring to research article that rely on empirical evidence. One book that time and time again I go back to refer is the Principles of Forecasting book by Armstrong and its website which I would recommend as an excellent read for any forecaster, provides great insight on usage and guiding principles of extrapolation methods.
To answer you first question - What I want to know is if this is unusual?
There is a chapter called Extrapolation for Time-Series and Cross-Sectional Data which also available free in the same website. The following is the quote from the chapter
"For example, in the real-time M2-competition, which examined 29 monthly series, Box-Jenkins proved to be one of the least-accurate methods and its overall median error was 17% greater than that for a naive forecast"
There lies an empirical evidence on why your mean forecasts was better than ARIMA models.
There is also been study after study in empirical competitions and the third M3 competition that show Box - Jenkins ARIMA approach fails to produce accurate forecast and lacks evidence that it performs better for univariate trend extrapolation.
There is also another paper and an ongoing study by Greene and Armstrong entitled "Simple Forecasting: Avoid Tears Before Bedtime" in the same website. The authors of the paper summarize as follows:
In total we identified 29 papers incorporating 94 formal comparisons of the accuracy of forecasts from complex methods with those from simple—but not in all cases sophisticatedly simple—methods. Eighty-three percent of the comparisons found that forecasts from simple methods were more accurate than, or similarly accurate to, those from complex methods. On average, the errors of forecasts from complex methods were about 32 percent greater than the errors of forecasts from simple methods in the 21 studies that provide comparisons of errors
To answer your third question: does this indicate that I've set something up wrong? No, I would aconsider ARIMA as complex method and Mean forecast as simple methods. There is ample evidence that simple methods like Mean forecast outperform complex methods like ARIMA.
To answer your second question: Does this mean the times series I'm using are strange?
Below are what I considered to be experts in real world forecasting:
All of the above researchers advocate, simplicity (methods like your mean forecast) vs. Complex methods like ARIMA. So you should feel comfortable that your forecasts are good and always favor simplicity over complexity based on empirical evidence. These researchers have all contributed immensely to the field of applied forecasting.
In addition to Stephan's good list of simple forecasting method. there is also another method called Theta forecasting method which is a very simple method (basically Simple Exponential smoothing with a drift that equal 1/2 the slope of linear regression) I would add this to your toolbox. Forecast package in R implements this method.
This is not at all surprising. In forecasting, you very often find that extremely simple methods, like
outperform more complex methods. That is why you should always test your methods against these very simple benchmarks.
A quote from George Athanosopoulos and Rob Hyndman (who are experts in the field):
Some forecasting methods are very simple and surprisingly effective.
Note how they explicitly say they will be using some very simple methods as benchmarks.
In fact, their entire free open online textbook on forecasting is very much recommended.
EDIT: One of the better-accepted forecast error measures, the Mean Absolute Scaled Error (MASE) by Hyndman & Koehler (see also here) measures how much a given forecast improves on the (in-sample) naive random walk forecast: if MASE < 1, your forecast is better than the in-sample random walk. You'd expect this to be an easily beaten bound, right?
Not so: sometimes, even the best out of multiple standard forecasting methods like ARIMA or ETS will only yield a MASE of 1.38, i.e., be worse (out-of-sample) than the (in-sample) random walk forecast. This is sufficiently disconcerting to generate questions here. (That question is not a duplicate of this one, since the MASE compares out-of-sample accuracy to in-sample accuracy of a naive method, but it is also enlightening for the present question.)
