2
$\begingroup$

I have two regression models: a linear model with one variable, and a neural network with many. I want to test whether the neural network is better than the linear model.

I am wondering the best way to do this. I think I could do leave-one-out cross validation with a Wilcoxon signed-rank test by comparing the out-of-bag errors from each model. Would this work? Is there any literature to support this approach?

Improve this question
$\endgroup$
11
  • $\begingroup$ I think you may using the term out-of-bag incorrectly. Neither linear models nor neural networks typically use bagging as a feature. $\endgroup$
    – mkt
    Commented Oct 13, 2019 at 7:04
  • $\begingroup$ It's not clear why you want to use signed ranks in your evaluation, though. k-fold cross validation will let you calculate the RMSE of the two models, which is a reasonable basis for comparison. $\endgroup$
    – mkt
    Commented Oct 13, 2019 at 7:06
  • $\begingroup$ @mkt I suppose I mean "out of sample" rather than "out of bag" $\endgroup$
    – rhombidodecahedron
    Commented Oct 13, 2019 at 7:10
  • $\begingroup$ @mkt How do I go from RMSE values to a hypothesis test? $\endgroup$
    – rhombidodecahedron
    Commented Oct 13, 2019 at 7:11
  • $\begingroup$ What is the hypothesis you are trying to test? $\endgroup$
    – mkt
    Commented Oct 13, 2019 at 7:13

1 Answer 1

Reset to default
2
$\begingroup$

Two compare two regression models (RM) using hypothesis testing two cases need to be considered:

Large data set S: One can divide S into several disjoints training sets and a single test set. Each RM is trained on each training set and then tested in the test set. An analysis of variance using the quasi-F test can be performed to test if RM1 is better than RM2.

Small data set S: Here one must resort to k-fold cross validation. This violates one of the assumptions of classical statistical tests, the problem is namely that each instance appears in more than one set. In this case one can use the 5 x 2 CV paired t test to compare the two RMs, this test is detailed in section 3.5 of "Approximate statistical tests for comparing supervised classification learning algorithms" by T. Diettrich. Link: https://www.mitpressjournals.org/doi/10.1162/089976698300017197

Another good source of information is: https://machinelearningmastery.com/statistical-significance-tests-for-comparing-machine-learning-algorithms/

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for this reply! The 5x2cv test is for classification algorithms; how can it be adapted for use with regression? $\endgroup$
    – rhombidodecahedron
    Commented Oct 15, 2019 at 2:40
  • $\begingroup$ The 5 x 2 CV method is not limited to classification, it is even the recommended method by Jason for regression in the website linked above (see the Q&As at the bottom of the website). The 5 x 2 CV method is "just" a clever way of doing cross validation so that the assumptions of the t test are valid. For the t test one can use the loss evaluated in the test set in the case of regression models. $\endgroup$
    – Javier-Acuna
    Commented Oct 15, 2019 at 6:18
  • $\begingroup$ thanks - is there a way of knowing whether the assumptions of the t test are valid for a particular problem at hand? $\endgroup$
    – rhombidodecahedron
    Commented Oct 15, 2019 at 6:29
  • $\begingroup$ One can check for normality and equality of variance but the assumption that the samples are iid is not possible to check only from data, one needs to know how the experiment was made. $\endgroup$
    – Javier-Acuna
    Commented Oct 15, 2019 at 7:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.