I am comparing the root mean squared error (RMSE) of different regression models that were fit to simulated data. I have RMSE from 30 different simulated data sets for each regression model. Some of the average RMSE values from each model are very similar. Normally, I would use a statistical hypothesis test or confidence intervals of the RMSE values to conclude whether there is a statistically signifcant difference in RMSE values. I have read that it is inappropriate to use statistical hypothesis testing on simulated data results, since you control the sample size and variance of the results. Therefore, in order to compare the RMSE of the models I am looking at other methods. I know empirical cumulative distribution function (ECDF) can be used for this purpose, as well as boxplots.

I was wondering if anyone knows of a less subjective method that can be used to compare the similarity of RMSE results?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.