7
$\begingroup$

I have question on evaluating forecasting software with expert systems:

Are there any objective assessment (not from the manufacturer) on expert forecasting software in the literature?

I found two references, but both are old:

  1. Principles of Forecasting (2001) Armstrong
  2. Forecasting methods and applications (1998) Makridakis et al.

SCA and AUTOBOX seem to be two expert systems in the market, are there any comparisons made on these two software? Are there any other software that have expert systems?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
6
$\begingroup$

The last review of SCA and AUTOBOX was done in 1995 (see here). We had approached Prof. Len Tashman the editor of Foresight for a head-to-head review of AUTOBOX vs SAS but SAS's representative replied in the negative in July 2010:

ARIMA modeling is an interesting subject, but not one I would want to focus on, or one I think of as central for modern forecasting methods. Even with the scope of the comparison explicitly limited to ARIMA functionality, there is still an imbalance in an Autobox versus SAS comparison. Autobox is a niche product that is all about automatic ARIMA modeling. On the other hand, SAS/ETS offers a broad range of tools for econometrics and time series modeling—our PROC ARIMA is just one of twenty-seven ETS procedures. For forecasting, our primary offering is SAS Forecast Server, not SAS/ETS. In turn, SAS/ETS and Forecast Server are just two of the dozens of offerings in the SAS product line. So even if the article is specifically limited to ARIMA, I’m afraid a “SAS” versus Autobox comparison would still come across a little like comparing a Notepad alternative to “Microsoft”.

To correct the above: AUTOBOX is and always has been about automatic and non-automatic modelling of both ARIMA and Transfer Functions. In this way experts can use their expertise much like SCA and SAS or at their option use the expert heuristics within AUTOBOX as a productivity aid.

There is a fairly recent review of AUTOBOX in 2010 (see here) that was very thorough.

I am one of the developers of AUTOBOX. In case you need more help please feel free to contact me and/or pose additional questions.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ @isrishstat thank you for detailed review. Is SCA also an expert system much like autobox with all the built in functionality like outlier detection, level shift etc ? $\endgroup$
    – forecaster
    Aug 25, 2013 at 0:03
  • $\begingroup$ @forecaster SCA,I believe handles Pulses & Steps/Level while AUTOBOX also includes Seasonal Pulses and Local Time Trends. AUTOBOX detects/remedies points in time where the pararameters and/or the variance of the errors change.Most importantly SCA does not reduce the model to only having significant structure.I would suggest that you select 10 time series problems (both ARIMA and TF) and submit them to both vendors and have them return all the details.Then share all results with both vendors and ask them to compare/critique what they did and what the competitor didn't do correctly. $\endgroup$
    – IrishStat
    Aug 25, 2013 at 11:59
  • $\begingroup$ @forecaster .... Continuing .... SCA has the capability of incorporating innovative outliers which (if present) AUTOBOX proxies with additive outliers. $\endgroup$
    – IrishStat
    Aug 25, 2013 at 14:15
11
$\begingroup$

The only proper evaluation of automatic forecasting software is in head-to-head competition on real data. The last large-scale competition like that was done in 2000 in the M3 competition. The results are publicly available: http://www.forecastingprinciples.com/paperpdf/Makridakia-The%20M3%20Competition.pdf. Draw your own conclusions about what automatic software is the best.

Software and automatic methods introduced since 2000 are not listed there. There are some additional comparisons published in the International Journal of Forecasting from time to time.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Good point. I would also like to point out that many recent improvements continue to be made in the heuristics underlying model identification so that any 13 year old competition results (from 1 and only 1 origin) may no longer be relevant. $\endgroup$
    – IrishStat
    Aug 25, 2013 at 11:50
  • $\begingroup$ Did AUTOBOX participate in M4? $\endgroup$
    – Pankaj Joshi
    May 13, 2019 at 9:52
  • $\begingroup$ I don't think so. The M4 results are publicly available here: doi.org/10.1016/j.ijforecast.2018.06.001 $\endgroup$
    – Rob Hyndman
    May 13, 2019 at 11:07
1
$\begingroup$

This is not an answer but rather a comment to @Irishstat. This is a lengthy comment so I'm writing this as an answer.

@Irishstat, I'm really surprised why an editor of a reputable journal like Foresight need to get permission from a SAS representative to do a comparison between two software. I would consider that this is a scientific empirical research that anyone who has access to SAS forecasting software and Autobox should be able to do, especially in universities. I was reading the golden rule of forecasting article by Professor Armstrong who recommends empirical evidence based adoption of methods, software/tools. I would think that a journal like Foresight, should be able to do an independent comparison of the forecasting software in the interest of forecasting community and not specifically tied to a forecasting vendor, provided adequate funding.

I have to applaud autobox and forecast pro for competing in the forecasting competition such as M3.

I know there is an M4 competition, I'm not sure if SAS competed in M4. Do we know if and when the M4 compettiion results will be published ?

Improve this answer
$\endgroup$
4
  • $\begingroup$ I agree with you that one shouldn't have to get permission to conduct an objective comparison BUT SAS is a major advertiser and perhaps that came into play. I should also add that the Foresite Journal Editor doesn't want to unnecessarily ruffle any feathers and historically there has been an unwritten rule about ruffling feathers. With respect to the M4 competition please see the AUTOBOX Users Group Blog linkedin.com/groups?home=&gid=2511642 entitled "The Sorry Story .... " $\endgroup$
    – IrishStat
    Jul 1, 2014 at 19:53
  • $\begingroup$ One other comment, The Foresite Journal Editor is a very ethical and professional forecaster. He just couldn't get SAS to agree to do something he thought was valuable to the community. It was a pity ! $\endgroup$
    – IrishStat
    Jul 1, 2014 at 20:42
  • $\begingroup$ my comments on forecast comparisons $\endgroup$
    – IrishStat
    Jul 2, 2014 at 9:39
  • $\begingroup$ M4 results were published in the IJF here: doi.org/10.1016/j.ijforecast.2018.06.001 $\endgroup$
    – Rob Hyndman
    Dec 21, 2018 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.