I'm trying to evaluate some software for forecast accuracy. It works by summing up all the orders from a number of locations for each month, then determines the best model out of a series of models based on the one the generates the minimum MSE. The it takes that model to forecast the demand for each location. For example, for Jan-Jun, Location A has demand (1,0,2,0,0,3) and Location B has demand (2,1,0,0,3,1). The aggregate would be A+B =(3,1,2,0,3,4). The software would then build models using ses, holt, MA, Croston's and Weighted Average. The one that produces the smallest MSE (in-sample) would be chosen to build the forecast for July. Then it would do the same thing again for August when it has an actual demand for July. It continues this way and may change the forecasting method at each month based on the minimum MSE. Therefore, it may generate forecasts for July-Dec using methods like, for example, (ses, ses, MA, Croston's ses, holt).
I currently have data from Jan 2016 to Dec 2017 (24 months) and I'm looking for advice regarding how to determine how well the tool determines a forecast. I thought about using tsCV, but that assume the same model will be applied each month in a rolling forecast, which isn't the case.