2
$\begingroup$

Let's assume that I have two regression models A and B, which are tested on the same dataset which contains N samples. Therefore, I can estimate the error of both model A and model B on each of the N instances from the dataset. My question in, how can I verify that the difference between the two models is statistically significant?

Furthermore, assuming that I can M such models which are tested on the same dataset which contains N samples, how can I verify that a certain model is statistically significantly better than the rest of the models?

Some information about the models:

  • The models are not nested.
  • They are completely different models.
  • The models don't share the predictors (one model uses more features), while they share the outcome (the target is the same).

Thank you for your help!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Are these models nested? Do they share outcome or predictors? Or, are they completely different models? It might be better if you could provide more information about the models. $\endgroup$
    – T.E.G.
    Commented May 5, 2020 at 14:19
  • 1
    $\begingroup$ I updated the question. My background is in Machine Learning, so I apologize upfront if the terminology is confusing :) $\endgroup$
    – gorjan
    Commented May 5, 2020 at 14:26

2 Answers 2

Reset to default
2
$\begingroup$

If you have a separate test data set (that you did use to tune and fit the model) then you can sample the performance of the models and with those samples compare whether the mean performances are significantly different (in a similar way as a t-test compares differences between two samples, or you use a Monte Carlo approach).

Improve this answer
$\endgroup$
7
  • $\begingroup$ I do have a separate test data set, that I did not use to fit the model. I fit the model on the training dataset and then I want to use the test data set just to perform inference with the models, and compare their performance. $\endgroup$
    – gorjan
    Commented May 5, 2020 at 15:06
  • $\begingroup$ Also, can you elaborate a bit more on should I measure whether the difference between the model is significant? According to the literature I read, I should probably be using the Welch's t-test, since I have equal sample sizes but different variances. $\endgroup$
    – gorjan
    Commented May 5, 2020 at 15:13
  • 1
    $\begingroup$ There are many ways to define performance and there are many ways to estimate and compare the performance of two models. What are your requirements (how do you wish to measure express the 'performance' and what other considerations play a role)? $\endgroup$
    – Sextus Empiricus
    Commented May 5, 2020 at 15:24
  • $\begingroup$ Well, the models performance is measured with the Mean Squared Error across the test set. For model A I get error a, while for model B I get error b. Additionally, a < b. I want to know whether the difference in the error is statistically significant so that I can claim that model A is significantly better than model B. Does that clarify things? In anyways, thank you for taking the time to help me! $\endgroup$
    – gorjan
    Commented May 5, 2020 at 15:29
  • $\begingroup$ In that case it is a bit similar to en.m.wikipedia.org/wiki/F-test_of_equality_of_variances comparing whether two chi-squared distributed variables (assuming your error is more or less normal distributed and with zero mean) are significantly different. But in your case you have non-central chi-squared distributed variables. I guess that this is a question on it's own and not easy (you may have the same mean squared error but with different underlying distribution of the error). The simple way out is to compute/estimate the sample distribution of mean squared error with a MC approach. $\endgroup$
    – Sextus Empiricus
    Commented May 5, 2020 at 15:48
0
$\begingroup$

A good model is the one that fulfills all the assumptions, for example, in the case of a linear regression, , there is normality, homocedasticity, independence, etc. If your two models meet the assumptions, one way to compare them can be AIC, BIC, adjusted R ^ 2. It is also necessary to consider the selection of variables, the most used methods are forward, stepwise and backward.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the answer. However, based on the literature review I've done, these methods won't tell me whether the difference between the models is statistically significant. Am I right? $\endgroup$
    – gorjan
    Commented May 5, 2020 at 15:22
  • $\begingroup$ Adjusted AIC, BIC, R ^ 2 are ways to compare models, but you're right, they don't tell you if the difference is statistically significant. The problem is that your models are not nested, otherwise, a hypothesis test could be done to see the difference of your models. (sorry for my English) $\endgroup$
    – CJ.Learns.Math
    Commented May 5, 2020 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.