I collected a sample of 1000 observation (time series data) with 100 predictors variables in order to predict another one. I want to use some prediction models but I know that, unfortunately, overfitting problem exist. So I split the sample, in and out, then, basically, I have to estimate parameters in sample and check the prediction quality out of sample against a benchmark; I use MSE and or related metrics. Until here I don't have great doubts.

Moreover I want to check for the presence of relevant non linearity in the link among predictors and predicted variable. The most flexible alternative seems me the Artificial Neural Network (ANN) models and I want to try with them. So starting from the same split above the idea is again to calibrate the ANN in sample and test it out of sample. As assumption I consider that no useless predictors exist and, after standardization, I use all of them. However in ANN there are a lot of parameters, firstly: number of hidden layers, number of nodes, type of activation function. It seem me that one possibility is to define a grid of models with combinations of parameters above and check the prediction quality out of sample looking for the best model in term of the same metrics used for regressions. I fear that this method is not the most used. However it is admissible ?

So I want to explore the Cross Validation (CV) method as remedy for overfitting but I have several doubt about that. It seems me right proceed as follow: I hold constant, as hyperparameters, the number of hidden layers, nodes and kind of activation function. So I can calibrate the model using K-fold strategy averaging parameters (weights in nodes). My questions:

1) Basically the procedure described above is CV in ANN? Or CV in ANN is different thing? In the first case have you some suggestion? In the second, basically, what I have to do?

2) CV can help me to find the best number of hidden layer and/or node?

3) Probably the fact that the data are in time series form is a problem for standard CV. What are the easier remedy?

4) Split the sample and use only in sample part is usual in CV, or it is made on all data?

  • $\begingroup$ One place to start is with the fact that a single-layer ANN without an activation function trained with MSE loss is identical to a simple linear regression model. It is possible to add complexity to your model slowly-ish by building up from this basic model. A first step might be, say, adding in a single hidden layer down to 2 outputs, a relu, then a final layer down to 1 output. This approximately doubles the number of variables in the model. A single hidden layer down to 3 outputs triples the number of variables. $\endgroup$ – Scott Sep 25 '19 at 15:34
  • $\begingroup$ When you say that you have a time series on 100 variables and a response variable, do you mean that you use the entire time series to make one prediction? An example would be Siri/Alexa using the entirety of your couple of seconds of spoken audio to predict the words you spoke. $\endgroup$ – Dave Sep 25 '19 at 16:34
  • $\begingroup$ @Dave I use the entire set of predictors, 100 variables (one vector of 100 numbers), to predict one variable (one number). Then I have 1000 obs for train and test the ANN. $\endgroup$ – markowitz Sep 26 '19 at 7:46
  • $\begingroup$ @markowitz What, then, is the time series component? $\endgroup$ – Dave Sep 26 '19 at 9:25
  • $\begingroup$ @Dave in regression form we have $y_t = \beta X_{t-1} + \epsilon_t$ $\endgroup$ – markowitz Sep 26 '19 at 10:16