I am comparing the root mean squared error (RMSE) of different regression models that were fit to simulated data. I have RMSE from 30 different simulated data sets for each regression model. Some of the average RMSE values from each model are very similar. Normally, I would use a statistical hypothesis test or confidence intervals of the RMSE values to conclude whether there is a statistically signifcant difference in RMSE values. I have read that it is inappropriate to use statistical hypothesis testing on simulated data results, since you control the sample size and variance of the results. Therefore, in order to compare the RMSE of the models I am looking at other methods. I know empirical cumulative distribution function (ECDF) can be used for this purpose, as well as boxplots.
I was wondering if anyone knows of a less subjective method that can be used to compare the similarity of RMSE results?