Timeline for Statistical significance between multiple regression models
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 5, 2020 at 16:30 | vote | accept | gorjan | ||
| May 5, 2020 at 16:30 | comment | added | gorjan | Thank you for the help. I will accept the answer as correct now. | |
| May 5, 2020 at 15:59 | comment | added | Sextus Empiricus | You could also use the Delta method. | |
| May 5, 2020 at 15:48 | comment | added | Sextus Empiricus | In that case it is a bit similar to en.m.wikipedia.org/wiki/F-test_of_equality_of_variances comparing whether two chi-squared distributed variables (assuming your error is more or less normal distributed and with zero mean) are significantly different. But in your case you have non-central chi-squared distributed variables. I guess that this is a question on it's own and not easy (you may have the same mean squared error but with different underlying distribution of the error). The simple way out is to compute/estimate the sample distribution of mean squared error with a MC approach. | |
| May 5, 2020 at 15:29 | comment | added | gorjan |
Well, the models performance is measured with the Mean Squared Error across the test set. For model A I get error a, while for model B I get error b. Additionally, a < b. I want to know whether the difference in the error is statistically significant so that I can claim that model A is significantly better than model B. Does that clarify things? In anyways, thank you for taking the time to help me!
|
|
| May 5, 2020 at 15:24 | comment | added | Sextus Empiricus | There are many ways to define performance and there are many ways to estimate and compare the performance of two models. What are your requirements (how do you wish to measure express the 'performance' and what other considerations play a role)? | |
| May 5, 2020 at 15:13 | comment | added | gorjan | Also, can you elaborate a bit more on should I measure whether the difference between the model is significant? According to the literature I read, I should probably be using the Welch's t-test, since I have equal sample sizes but different variances. | |
| May 5, 2020 at 15:06 | comment | added | gorjan | I do have a separate test data set, that I did not use to fit the model. I fit the model on the training dataset and then I want to use the test data set just to perform inference with the models, and compare their performance. | |
| May 5, 2020 at 15:03 | history | answered | Sextus Empiricus | CC BY-SA 4.0 |