I am trying to build an R tool for forecasting a (hopefully) wide range of time-series. I have settled on using several models, taking the forecasts from each, and deriving a weighed average of them using some weights.
My approach for arriving at appropriate weights for the averaging is to evaluate each model several times on parts of the historical data. For example, for monthly series I do the following:
I evaluate a one-step forecast for each model (five of them) for each of the last 12 months in the historical data $\{a_{i,j}\mid i\in\{1,\ldots,5\},j\in\{1,\ldots,12\}\}$ with $a'_{j}$ the actual observations. I evaluate six non-overlapping (ex. Oct+Nov+Dec, then Jul+Aug+Sep, etc.) three-step forecasts for each model, taking the mean of the forecasts for each of the five models at a time $\{b_{i,j}\mid i\in \{1,\ldots,5\},j\in\{1,\ldots,6\}\}$ with $b'_{j}$ as the mean of the relevant actuals at each time. Finally, I evaluate four six-month-overlapping (ex. Jan through Dec, Jul through Jun, etc.) 12-step forecasts for each model, taking again the mean for each model, getting the final set $\{c_{i,j}\mid i\in\{1,\ldots,5\},j\in\{1,\ldots,4\}\}$ with $c'_{j}$ the means of the relevant actuals.
I put
$$A=\left(\begin{array}{c}a_{i,j}\\b_{i,j}\\c_{i,j}\end{array}\right), x=\left(\begin{array}{c}w_1\\\ldots\\w_5\end{array}\right), b=\left(\begin{array}{c}a'_j\\b'_j\\c'_j\end{array}\right)$$ ! and use
optim
from thestats
package to optimise $x$ to give the least MAE between the two vectors $Ax$ and $b$.
So my question is
Is this approach conceptually valid, considering that this evaluates something like the whether the model procedure is approriate for the time series, and not whether a particular model-with-parameters is?
EDIT: Question paraphrased significantly to focus on aspects not answered here.