Test which one of two (mechanistic / non-regression) models fits the observations better

Question

There's a large number of similar questions, but they mostly seem to be about regression models and deal with cross validation, comparison of nested statistical models via likelihood-ratio, etc. Apologies if these methods are applicable and I just don't realize how.

Problem: I want to compare two relatively complex simulation models which predict a time-series and test which one fits the observations better. No parameters are being fitted. These are not regression models.

I'd like to say whether or not the accuracy of the two models is significantly different, so a simple comparison of RMSEs isn't enough.

Example: Let's say I want to test which one of two numerical weather prediction models is better at predicting the temperature at place X (where I have a thermometer). Both models are mostly identical and rely on the same input variables, except for one process, which is modelled in a fundamentally different way (i.e. this is not just about adding a single parameter).

What kind of test would be suitable for this? I'm thinking about testing for differences in the residuals, but I'm not sure in what way exactly (i.e. do I test for differences in the residuals and just check which distribution is closer to zero? Do I compare absolute or squared residuals with a KS-Test? If yes, which one?). Or is there a simpler solution?

(Bonus: In my specific problem I have a strong suspicion which of the two models is more accurate, so a one-sided variant would be interesting.)

Answer 1

I agree that running statistical tests/fitted statistical models on the residuals is the right way to go.

Bias: which set of residuals has a mean closer to zero? Use a t-test: You probably want to use the Welch/Satterthwaite version that allows for the possibility that the variances are different in each group. t-tests are fairly robust to non-Normality (e.g. see answers to this question).

Variance: Levene's test or Levene's median test (see here).

RMSE: I'm not sure, but any sensible procedure to test the difference in locations (means/medians) of the distribution of squared errors (which incorporate both variance and bias) [t-test, Mann-Whitney, etc.].

More parametrically, you could fit a generalized least-squares model that allowed for changes in mean and variance of residuals across groups. In R's nlme package that would be something like

gls(resid ~ resid_type, weights=varIdent(form=~1|resid_type), data=...)

This would also allow you to ask deeper/more interesting questions, e.g. whether the bias and variance vary across the data set (e.g. bias or variance change with temperature, or with other covariates).

A Kolmogorov-Smirnov test will just test whether the residuals are different, which doesn't seem like it will be very interesting ...