1
$\begingroup$

How to test the difference between prediction accuracy of two regression models?

My idea is to compare the errors of the two models, e.g., one-predictor vs. multiple-predictor model, in order to show that the difference between the two models (i.e., in terms of the accuracy of their results) is statistically significant. I would like to compare the differences, e.g., with a paired t-test procedure.

The problem is different from both:

  1. Classic statistical significance between means. I.e., the way I want to implement this to test the errors is the same, but the tested entities differs (classic statistical significance testing: means of measurements; regression models: errors of the regression model).
  2. Machine learning classification problem (see, e.g., Statistical significance when comparing two models for classification) -- here I analyze a regression model, not a classifier (i.e., no cross validation) (regression: quantitative parameter is utilized to predict a quantitative output; classification: quantitative parameters are used to predict a class category). Maybe my regression problem can be stated as a logistic regression, or something -- I did not yet analyzed this, but your feedback is welcome.

I also thought about the common mathematical formulation for ANOVA (where statistical significance is usually tested), and the GLM, but I do not think this resemblance can be beneficial when planning to test the models between each other, but I may be wrong.

Any thoughts on the subject would be much appreciated. I am sorry for any inconsistencies in my question, I tried to be as specific and possible.

$\endgroup$
0

0