I collected a sample of 1000 observation (time series data) with 100 predictors variables in order to predict another one. I want to use some prediction models but I know that, unfortunately, overfitting problem exist. So I split the sample, in and out, then, basically, I have to estimate parameters in sample and check the prediction quality out of sample against a benchmark; I use MSE and or related metrics. Until here I don't have great doubts.
Moreover I want to check for the presence of relevant non linearity in the link among predictors and predicted variable. The most flexible alternative seems me the Artificial Neural Network (ANN) models and I want to try with them. So starting from the same split above the idea is again to calibrate the ANN in sample and test it out of sample. As assumption I consider that no useless predictors exist and, after standardization, I use all of them. However in ANN there are a lot of parameters, firstly: number of hidden layers, number of nodes, type of activation function. It seem me that one possibility is to define a grid of models with combinations of parameters above and check the prediction quality out of sample looking for the best model in term of the same metrics used for regressions. I fear that this method is not the most used. However it is admissible ?
So I want to explore the Cross Validation (CV) method as remedy for overfitting but I have several doubt about that. It seems me right proceed as follow: I hold constant, as hyperparameters, the number of hidden layers, nodes and kind of activation function. So I can calibrate the model using K-fold strategy averaging parameters (weights in nodes). My questions:
1) Basically the procedure described above is CV in ANN? Or CV in ANN is different thing? In the first case have you some suggestion? In the second, basically, what I have to do?
2) CV can help me to find the best number of hidden layer and/or node?
3) Probably the fact that the data are in time series form is a problem for standard CV. What are the easier remedy?
4) Split the sample and use only in sample part is usual in CV, or it is made on all data?