2
$\begingroup$

I've estimated 2 models on simulated data and basically replicate the estimations 100 times. What I want to see is if the models are actually different in terms of their MSE. What I've done is to keep the MSE1 and MSE2 (for model 1 and 2) for each one of the 100 simulations, and I was thinking of comparing the mean MSE (I am not sure if this is actually the correct way of doing it). Anyway I don't think I can apply a t-test due to the fact that the variables MSE1 and MSE2 are not part of a normal population (actually they look like a sort of $\chi^2$) because it's truncated ,no negative MSE are allowed.

Should I apply a $t$ test (isn't normality of observations an assumption of this test?) or is there any other method I can use to compare the both models? Thanks in advance.

$\endgroup$
  • 1
    $\begingroup$ The distribution of the data does not matter, the residuals are what's important. Anyway, I would not do this. What kind of models are these? $\endgroup$ – user2974951 Feb 19 at 14:52
  • $\begingroup$ If you want to test the differences on the means for two different groups, aren't you assuming that the variable of interest is actually normal? I was pretty sure it was this way. In any case I am working with spline regressions, I am testing if placing the knots adaptively leads to an smaller MSE than placing knots uniformly. That's why I am trying to compare the MSE's mean for both models $\endgroup$ – RScrlli Feb 19 at 14:56
  • 2
    $\begingroup$ That is a common misconception, a two sample t-test makes no assumptions on the variable distributions. Since you have linear models it would be better to compare them with a log likelihood score, you can even use an ANOVA. $\endgroup$ – user2974951 Feb 19 at 15:08
  • $\begingroup$ Okey I get it, actually the mean of the variable in consideration should be normally distributed. If the variable itself is normally distributed things get easier cause we have that its mean is also normally distributed. Thanks for pointing that out. I'll go for the log likelihood score $\endgroup$ – RScrlli Feb 19 at 15:15
  • $\begingroup$ The mean will be normally distributed, regardless of the underlying distribution, provided the sample size is sufficient and the underlying distribution has finite mean and variance, thanks to the central limit theorem. $\endgroup$ – Robert Long Feb 19 at 15:57
1
$\begingroup$

You can use the log likelihood score to compare different nested models, here is an example in R with two polynomial models using an ANOVA for comparison.

> mod1=lm(mpg~poly(disp,2)+hp+cyl,data=mtcars)
> mod2=lm(mpg~poly(disp,3)+hp+cyl,data=mtcars)
> anova(mod1,mod2)

Analysis of Variance Table

Model 1: mpg ~ poly(disp, 2) + hp + cyl
Model 2: mpg ~ poly(disp, 3) + hp + cyl
  Res.Df   RSS Df Sum of Sq      F    Pr(>F)    
1     27 222.5                                  
2     26 106.1  1    116.41 28.526 1.369e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

If these models are not nested than a (repeated) CV procedure could be used to estimate the better model (for ex. using MSE). However, I would not use a statistical test to compare the two models. A subjective decision based on the mean MSE values and considering the simplest model would be my choice.

Nevertheless, if you are so inclined you could use a t-test to compare the two samples with MSE values. If the assumptions do not hold you can still use a non-parametric test, which does not require a specific distribution of the data.

$\endgroup$
1
$\begingroup$

For a large population (your 100 iterations are enough in my opinion), sample mean will converge to a normal distribution, even for data that is not normally distributed! You can perform hypothesis testing on it using a T-test

However, the previous solution does all the work for you

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.