Best approach for selecting averaging weights

Question

I am trying to build an R tool for forecasting a (hopefully) wide range of time-series. I have settled on using several models, taking the forecasts from each, and deriving a weighed average of them using some weights.

My approach for arriving at appropriate weights for the averaging is to evaluate each model several times on parts of the historical data. For example, for monthly series I do the following:

I evaluate a one-step forecast for each model (five of them) for each of the last 12 months in the historical data $\{a_{i,j}\mid i\in\{1,\ldots,5\},j\in\{1,\ldots,12\}\}$ with $a'_{j}$ the actual observations. I evaluate six non-overlapping (ex. Oct+Nov+Dec, then Jul+Aug+Sep, etc.) three-step forecasts for each model, taking the mean of the forecasts for each of the five models at a time $\{b_{i,j}\mid i\in \{1,\ldots,5\},j\in\{1,\ldots,6\}\}$ with $b'_{j}$ as the mean of the relevant actuals at each time. Finally, I evaluate four six-month-overlapping (ex. Jan through Dec, Jul through Jun, etc.) 12-step forecasts for each model, taking again the mean for each model, getting the final set $\{c_{i,j}\mid i\in\{1,\ldots,5\},j\in\{1,\ldots,4\}\}$ with $c'_{j}$ the means of the relevant actuals.

I put

$$A=\left(\begin{array}{c}a_{i,j}\\b_{i,j}\\c_{i,j}\end{array}\right), x=\left(\begin{array}{c}w_1\\\ldots\\w_5\end{array}\right), b=\left(\begin{array}{c}a'_j\\b'_j\\c'_j\end{array}\right)$$ ! and use optim from the stats package to optimise $x$ to give the least MAE between the two vectors $Ax$ and $b$.

So my question is

Is this approach conceptually valid, considering that this evaluates something like the whether the model procedure is approriate for the time series, and not whether a particular model-with-parameters is?

EDIT: Question paraphrased significantly to focus on aspects not answered here.

Answer 1

I'm not sure if I'm following what you are asking, so I'll do my best to answer the question. It appears that what you are after is selecting weights based on time series cross validation or rolling forecast and using regression coefficients as weights on the hold out data. Check out Diebold's forecasting book and a chapter on forecasting combination. What you may be after is called time varying combining weights. See the below for a section from the above referenced book.

If you have more than 3 or 4 methods to combine then you are better off using simple averages as opposed to optimal weights. You could also use trimmed means or Winsorized means to remove outliers in place of simple averages. Several research actually supports using averages, which can be thought of an extreme form of regularization or shrinkage estimators.

Check out these two articles and several references in those articles.

1. Combining forecast by Scott Armstrong in Principles of forecasting.
2. Simple robust averages of forecasts: Some empirical results by Jose and Winkler