Expert forecasting software evaluation I have question on evaluating forecasting software with expert systems:
Are there any objective assessment (not from the manufacturer) on expert forecasting software in the literature? 
I found two references, but both are old:


*

*Principles of Forecasting (2001) Armstrong

*Forecasting methods and applications (1998) Makridakis et al.


SCA and AUTOBOX seem to be two expert systems in the market, are there any comparisons made on these two software?  Are there any other software that have expert systems?
 A: The last review of SCA and AUTOBOX was done in 1995 (see here). We had approached Prof. Len Tashman the editor of Foresight for a head-to-head review of AUTOBOX vs SAS but SAS's representative replied in the negative in July 2010: 

ARIMA modeling is an interesting subject, but not one I would want to
  focus on, or one I think of as central for modern forecasting methods.
  Even with the scope of the comparison explicitly limited to ARIMA
  functionality, there is still an imbalance in an Autobox versus SAS
  comparison. Autobox is a niche product that is all about automatic
  ARIMA modeling. On the other hand, SAS/ETS offers a broad range of
  tools for econometrics and time series modeling—our PROC ARIMA is just
  one of twenty-seven ETS procedures. For forecasting, our primary
  offering is SAS Forecast Server, not SAS/ETS. In turn, SAS/ETS and
  Forecast Server are just two of the dozens of offerings in the SAS
  product line. So even if the article is specifically limited to ARIMA,
  I’m afraid a “SAS” versus Autobox comparison would still come across a
  little like comparing a Notepad alternative to “Microsoft”. 

To correct the above: AUTOBOX is and always has been about automatic and non-automatic modelling of both ARIMA and Transfer Functions. In this way experts can use their expertise much like SCA and SAS or at their option use the expert heuristics within AUTOBOX as a productivity aid.
There is a fairly recent review of AUTOBOX in 2010 (see here) that was very thorough.
I am one of the developers of AUTOBOX. In case you need more help please feel free to contact me and/or pose additional questions.
A: The only proper evaluation of automatic forecasting software is in head-to-head competition on real data. The last large-scale competition like that was done in 2000 in the M3 competition. The results are publicly available: http://www.forecastingprinciples.com/paperpdf/Makridakia-The%20M3%20Competition.pdf. Draw your own conclusions about what automatic software is the best.
Software and automatic methods introduced since 2000 are not listed there. There are some additional comparisons published in the International Journal of Forecasting from time to time.
A: This is not an answer but rather a comment to @Irishstat. This is a lengthy comment so I'm writing this as an answer.
@Irishstat, I'm really surprised why an editor of a reputable journal like Foresight need to get permission from a SAS representative to do a comparison between two software. I would consider that this is a scientific empirical research that anyone who has access to SAS forecasting software and Autobox should be able to do, especially in universities. I was reading the golden rule of forecasting article by Professor Armstrong who recommends empirical evidence based adoption of methods, software/tools. I would think that a journal like Foresight, should be able to do an independent comparison of the forecasting software in the interest of forecasting community and not specifically tied to a forecasting vendor, provided adequate funding.
I have to applaud autobox and forecast pro for competing in the forecasting competition such as M3.
I know there is an M4 competition, I'm not sure if SAS competed in M4. Do we know if and when the M4 compettiion results will be published ?  
