I collected a sample of 600 observation (time series data) with 100 predictors variables in order to predict another one. I want to use some prediction models but I know that, unfortunately, overfitting problem exist; especially with many predictors. So I split the sample, in and out (300 vs 300), then, basically, I have to estimate models/parameters in sample and check the prediction quality out of sample against a benchmark; I use MSE and or related metrics. Until here I don't have great doubts.
I starting with regression and I used a variables selection rule with strategy like this: overfitting and selection model.
Then, I want to check for the presence of relevant non linearity in the links among predictors and predicted variable. The most flexible alternative seems me the Artificial Neural Network (ANN) models and I want to try with them. So starting from the same split above the idea is again to calibrate the ANN in sample and test it out of sample. Here I assume that no useless predictors exist (no variables selection). So, after standardization (semi-standardization for out-of-sample data), I use all of them. However in ANN there are several unknown hyperparameters that make the things complex in estimation terms. Firstly: number of hidden layers, number of nodes, type of activation function. To choose them exogenously is one possibility but seems me too strong assumption. It seem me that another possibility is to split again the "in sample data" in training and validation set (200 vs 100). Then, I define a "grid of models" with different hyperparameters. I calibrate any ANN model in training set and I check the prediction performance of them in validation set. The performance is measured with the same metrics used for, final, out of sample scrutiny. The word validation give the idea that the choice come from cross validation (CV). For dependent data, as usually time series are, the usual CV methods are inadequate but some ad hoc version exist (see here: Cross-validation techniques for time series data Choosing inner cross validation strategy for modeling time series data Using k-fold cross-validation for time-series model selection ). All these strategy bring us to obtain several split, even if much less than the standard ways. In my case I consider only one split: ($t_{1},…,t_{200}$) for training and ($t_{201},…,t_{300}$) for validation. Therefore it seems me that, what I made is not a true CV procedure.
Only the best ANN model, recalibrated on in sample data (training + validation = 300) and the best regression one achieve the final scrutiny based on out of sample data (300 unseen before).
This general strategy make sense? Some weak points?
The procedure described for ANN hyperparameters choice, among a grid of models/specifications, represent a proper method?
In the out of sample scrutiny I have only one realization for generalized error of the two models. I measure the related $MSE$ and or others metrics. Compare them it's enough in order to choose the best one?
EDIT: The AJKOER answer suggest me to give you some information more. I said that my data have time series form, these are weak stationary or at least I suppose it. Otherwise usual inference from only one realization (what I have) is not possible. More precisely I try to predict/forecast a series of bond yield taken in first difference. At first glance this series is not far from bond or stock returns even if is negatively related with the former. However the procedure that I follow is quite general and I can use it also for return of any financial assets. It is well known that all of them are hard to forecast successfully.
Just for information, in my experience with ARMA models, even if some interesting results appear in ACF or PACF, the out of sample forecast performance, usually, is very poor. White noise benchmark seems me almost unbeatable. Moreover we have to note that, in pure forecast, multivariate models like VARs show usually worse performance that univariate models. For this reason I leave prediction models that consider only past own realizations and move to more general approach; more close to what predictive learning literature suggest.
Moreover I embrace the distinction between prediction and causation. Here I follow a purely predictive approach. Read here for more detail (Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict") Structural models and relationship (statistical associations) Endogeneity in forecasting Paradox in model selection (AIC, BIC, to explain or to predict?) Regression: Causation vs Prediction vs Description). I want to built a pure data driven model, therefore any theoretical points are not considered; extremely view, I know but that's it. I aware that about bond return predictability, term premia, term structure, ecc, wide and relevant literature exist but I want precisely do not consider it here.
