0
$\begingroup$

I am comparing the predictive power of two regression models. RMSE gain is $13\%$ using a full model compared to reduced one on the test set. I do not want to compromise on model complexity hence wondering if $13\%$ is significant enough to use full model.

Correction: RMSE for full model is $.13$ whereas for reduced one is $.15$. Any feedback would be highly appreciated.

Improve this question
$\endgroup$
19
  • 1
    $\begingroup$ If the two models are nested, you should run an F-test. $\endgroup$
    – stans
    Dec 6, 2022 at 4:35
  • $\begingroup$ Full model have 5 extra variables compared to the reduced one. So yes they are nested. Is there any other test that I should run as well? $\endgroup$
    – Arti Thakur
    Dec 6, 2022 at 4:43
  • $\begingroup$ No other test. F-test will answer your exact question: which of the two models is preferable. Of course, the assumptions have to be met. Either normality of residuals or sufficiently large sample size. $\endgroup$
    – stans
    Dec 6, 2022 at 5:11
  • 1
    $\begingroup$ But why do you want model parsimony? Yes, throwing tons of variables at a problem until the fit is good does put you at a risk of overfitting, but that does not assure you of overfitting (and leaving out variables puts you at risk of underfitting). I’m not saying it’s wrong to seek model parsimony, but it is not obvious to me why it must be a goal. $\endgroup$
    – Dave
    Dec 15, 2022 at 11:59
  • 1
    $\begingroup$ Do those five variables not make theoretical sense? $\endgroup$
    – Dave
    Dec 16, 2022 at 14:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.