I have been carrying out some analysis using Python where I have use OLS regression on a bunch of variables to see how they relate to the target. I have used the
Statsmodels package in Python which gives you a bunch of hypothesis (p(F), p(t)), criterion (AIC, BIC) and other statistical information (R2, Adj R2, RMSE) relating to the model.
I have used the same bunch for Neural Network Regression using SciKit Learn. I have used Cross validation to select the best ANN model for each combination using the RMSE score. However, these models do not give the other model selection options, such as hypothesis o criterion ratings.
Is it a valid approach to use the statistics from the OLS regression and use them to remove mdels from the ANN? So, for example I have the following model in OLS
y ~ x1 + x2 + x3
The resultant p(F) is 0.3, indicating the fit is not statistically significant. I would then ignore that model and move to the next one. Since the Neural Network models are created using the same variable combinations, can I also assume that will also be statistically insignificant and remove it from my list of models?