Model Selection between Classical Regression and Neural Network

Question

I have been carrying out some analysis using Python where I have use OLS regression on a bunch of variables to see how they relate to the target. I have used the Statsmodels package in Python which gives you a bunch of hypothesis (p(F), p(t)), criterion (AIC, BIC) and other statistical information (R2, Adj R2, RMSE) relating to the model.

I have used the same bunch for Neural Network Regression using SciKit Learn. I have used Cross validation to select the best ANN model for each combination using the RMSE score. However, these models do not give the other model selection options, such as hypothesis o criterion ratings.

Is it a valid approach to use the statistics from the OLS regression and use them to remove mdels from the ANN? So, for example I have the following model in OLS

y ~ x1 + x2 + x3

The resultant p(F) is 0.3, indicating the fit is not statistically significant. I would then ignore that model and move to the next one. Since the Neural Network models are created using the same variable combinations, can I also assume that will also be statistically insignificant and remove it from my list of models?

Answer 1

A neural network will typically allow for non-linearities and interactions that a linear regression model with just the main effects in the model cannot capture. In case interactions or non-linear effects are important, you cannot rely on the linear regression model to tell you about what may matter in an neural network.

Additionally, note that relying on statistical significance to make decisions about what model to use is fraught with issues. It is generally not a good idea, if you wish to achieve good out of sample predictions e.g. in terms of RMSE. In fact, it is a completely terrible approach (unless you somehow take the model selection into account, e.g. by bootstrapping the data repeatedly, selecting a model on each bootstrap sample and then averaging the predictions).