Need advice on evaluating forecast accuracy in R I'm trying to evaluate some software for forecast accuracy.  It works by summing up all the orders from a number of locations for each month, then determines the best model out of a series of models based on the one the generates the minimum MSE.  The it takes that model to forecast the demand for each location.  For example, for Jan-Jun, Location A has demand (1,0,2,0,0,3) and Location B has demand (2,1,0,0,3,1).  The aggregate would be A+B =(3,1,2,0,3,4).  The software would then build models using ses, holt, MA, Croston's and Weighted Average.  The one that produces the smallest MSE (in-sample) would be chosen to build the forecast for July.  Then it would do the same thing again for August when it has an actual demand for July.  It continues this way and may change the forecasting method at each month based on the minimum MSE.  Therefore, it may generate forecasts for July-Dec using methods like, for example, (ses, ses, MA, Croston's ses, holt).
I currently have data from Jan 2016 to Dec 2017 (24 months) and I'm looking for advice regarding how to determine how well the tool determines a forecast.  I thought about using tsCV, but that assume the same model will be applied each month in a rolling forecast, which isn't the case.
 A: First off, don't use the in-sample accuracy to choose a model. This will invariably lead to overfitting. In-sample accuracy is not a good guide to out-of-sample prediction. Instead, use a holdout sample.
Regarding your main question: again, use a holdout sample to see how well your algorithm performs on truly new data.
Thus, if you are interested in $h$-month-ahead forecasts:


*

*Fit your models to the data except for the last $2h$ months.

*Forecast all of them out to a horizon of $h$ months. Note the forecast error of each model, using rmse or whatever.

*Pick the model that performed best. Re-fit this model to the data except the last $h$ months. Forecast $h$ months ahead. Note the forecast error.


Do this for all your time series. Check how well this algorithm worked, and compare it to the performance of a few very simple benchmark methods, like always forecasting the historical mean, or the last observation. Or taking the average of all your candidate models' forecasts - averages of forecasts often outperform choosing the "best" method by some criterion.
