Hypothesis testing with regression models I have two regression models: a linear model with one variable, and a neural network with many. I want to test whether the neural network is better than the linear model. 
I am wondering the best way to do this. I think I could do leave-one-out cross validation with a Wilcoxon signed-rank test by comparing the out-of-bag errors from each model. Would this work? Is there any literature to support this approach? 
 A: Two compare two regression models (RM) using hypothesis testing two cases need to be considered:
Large data set S: One can divide S into several disjoints training sets and a single test set. Each RM is trained on each training set and then tested in the test set. An analysis of variance using the quasi-F test can be performed to test if RM1 is better than RM2.
Small data set S: Here one must resort to k-fold cross validation. This violates one of the assumptions of classical statistical tests, the problem is namely that each instance appears in more than one set. In this case one can use the 5 x 2 CV paired t test to compare the two RMs, this test is detailed in section 3.5 of "Approximate statistical tests for comparing supervised classification learning algorithms" by T. Diettrich. Link: https://www.mitpressjournals.org/doi/10.1162/089976698300017197
Another good source of information is: https://machinelearningmastery.com/statistical-significance-tests-for-comparing-machine-learning-algorithms/
