I have 2 different regression algorithms trained on the same dataset. I want to know if on a test set one of them performs significantly better than the other -- or at least if they are significantly different. What test do I use ? I am looking for something similar to McNemar's test that exists for classification algorithm.
2
-
1$\begingroup$ Please explain more about your regression algorithm. How does it produce a classification? $\endgroup$– AnikoCommented Jan 23, 2014 at 19:57
-
$\begingroup$ By "similar to McNemar's test" do you mean you are looking for a paired test? $\endgroup$– cbeleitesCommented Jan 23, 2014 at 23:38
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Since you have a test set at hand, you can do formal comparisons like this.
What you need is some reasonable loss function that measures the prediction error for each observation in the test set. For classification, this could be the 0-1 loss (1 if wrong, 0 if correct) and the squared loss for regression.
Then you can compare losses from the test set across the two available models. Usually, descriptive statistics are sufficient at this step, but it is not forbidden to supplement the descriptive comparison by some statistical test for the null hypothesis "true average loss is the same".
To do so, the test set must be considered a random sample from the population you want to apply the models to. A further, critical assumption for the standard paired tests (McNemar, t-test, Signed-Rank test, median test) to be valid is that the differences between the two losses have the same distribution for all observations in the test set. In a regression setting, with the squared loss, this is satisfied if the error terms in both models have constant variance.
