Neural Network vs regression in prediction I collected a sample of 600 observation (time series data) with 100 predictors variables in order to predict another one. I want to use some prediction models but I know that, unfortunately, overfitting problem exist; especially with many predictors. So I split the sample, in and out (300 vs 300), then, basically, I have to estimate models/parameters in sample and check the prediction quality out of sample against a benchmark; I use MSE and or related metrics.  Until here I don't have great doubts.
I starting with regression and I used a variables selection rule with strategy like this: overfitting and selection model.
Then, I want to check for the presence of relevant non linearity in the links among predictors and predicted variable. The most flexible alternative seems me the Artificial Neural Network (ANN) models and I want to try with them.
So starting from the same split above the idea is again to calibrate the ANN in sample and test it out of sample. Here I assume that no useless predictors exist (no variables selection). So, after standardization (semi-standardization for out-of-sample data), I use all of them.
However in ANN there are several unknown hyperparameters that make the things complex in estimation terms. Firstly: number of hidden layers, number of nodes, type of activation function. To choose them exogenously is one possibility but seems me too strong assumption.
It seem me that another possibility is to split again the "in sample data" in training and validation set (200 vs 100). Then, I define a "grid of models" with different hyperparameters. I calibrate any ANN model in training set and I check the prediction performance of them in validation set. The performance is measured with the same metrics used for, final, out of sample scrutiny.
The word validation give the idea that the choice come from cross validation (CV). For dependent data, as usually time series are, the usual CV methods are inadequate but some ad hoc version exist (see here: Cross-validation techniques for time series data Choosing inner cross validation strategy for modeling time series data Using k-fold cross-validation for time-series model selection ). All these strategy bring us to obtain several split, even if much less than the standard ways. In my case I consider only one split: ($t_{1},…,t_{200}$) for training and ($t_{201},…,t_{300}$) for validation. Therefore it seems me that, what I made is not a true CV procedure.
Only the best ANN model, recalibrated on in sample data (training + validation = 300) and the best regression one achieve the final scrutiny based on out of sample data (300 unseen before).

*

*This general strategy make sense? Some weak points?


*The procedure described for ANN hyperparameters choice, among a grid of models/specifications, represent a proper method?


*In the out of sample scrutiny I have only one realization for generalized  error of the two models. I measure the related $MSE$ and or others metrics. Compare them it's enough in order to choose the best one?
EDIT: The AJKOER answer suggest me to give you some information more. I said that my data have time series form, these are weak stationary or at least I suppose it. Otherwise usual inference from only one realization (what I have) is not possible. More precisely I try to predict/forecast a series of bond yield taken in first difference. At first glance this series is not far from bond or stock returns even if is negatively related with the former. However the procedure that I follow is quite general and I can use it also for return of any financial assets. It is well known that all of them are hard to forecast successfully.
Just for information, in my experience with ARMA models, even if some interesting results appear in ACF or PACF, the out of sample forecast performance, usually, is very poor. White noise benchmark seems me almost unbeatable.  Moreover we have to note that, in pure forecast, multivariate models like VARs show usually worse performance that univariate models. For this reason I leave prediction models that consider only past own realizations and move to more general approach; more close to what predictive learning literature suggest.
Moreover I embrace the distinction between prediction and causation. Here I follow a purely predictive approach. Read here for more detail (Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict") Structural models and relationship (statistical associations) Endogeneity in forecasting Paradox in model selection (AIC, BIC, to explain or to predict?) Regression: Causation vs Prediction vs Description).
I want to built a pure data driven model, therefore any theoretical points are not considered; extremely view, I know but that's it. I aware that about bond return predictability, term premia, term structure, ecc, wide and relevant literature exist but I want precisely do not consider it here.
 A: Here is my ideal opinion on a valid/rational course of action:
Step 1:  identify the realm that you're operating in, whether it be economics,  physical chemistry, …
Step 2: Based on Step 1, postulate all the applicable physical laws and generating processes that are likely significant drivers.  This may require research/consultation with an economist, physical chemists,... etc.
Step 3:  Build hypothetical simple models or inter-related models that are based on the identified generating process. Note: models and generating processes are based on the real-world opinion/research of experts in the particular arena.
Step 4: Populate said models/structures with generated data based on appropriate parent distributions via Monte Carlo methods. Start by selecting a very low level or noise (including associated inter-correlation noise structure, etc.).
Step 5: Investigate analytical tools you have available to develop parameter estimates assuming that you actually have roughly or precisely a correct singular model, or with a no model scenario, where a model-free approach is applicable. Repeat with multiple or shifting models as deemed appropriate in the realm of study.
This exercise will quickly reveal the power or lack thereof of analytical tools and associated methods applied to the various targetted arenas (so, you can stop wasting your time and effort in likely difficult reverse modeling situations).
Also, it may further, in failing, give real guidance on robust estimates of noise so that even naive poorly performing models may render rational error prediction bounds.
In essence, I am recommending expert base knowledge of a field to replicate data and then test applicable paths to extract the structure and assess error levels (if even possible), which are, in effect, all known with precision.
A: I don't disagree with AJKOER's answer, although you may not be in a situation in which you have much background knowledge that can be used in this way. Anyway, directly to your questions:

*

*This sounds OK. The problem obvsiously is that 300 points isn't that much for making all these decisions by optimisation, so you may be disappointed with the out of sample results of what comes out as optimal on your "in sample". However, I don't really know a good cure. In fact the only alternative is to make stronger assumptions, and then it depends on background knowledge whether you can justify these. Knowing something in advance and using it doesn't waste degrees of freedom in comparisons, but of course if you don't actually have some good knowledge backing up decisions, that may do harm rather than help. So I see why you're suggesting it as you suggest, and I have done things in this way, too. Let's say it may pay off to think a bit about what you can just decide in advance, because 300 observations will be very few if you want to make 100 decisions based on them.


*Imprecise question. I'd call it "some sort of cross-validation", but then I'm not sure about your definition of "some sort", and does it matter how one can call it?


*You're only comparing two models there with no further tuning, so I'd say this is OK.
