There's a large number of similar questions, but they mostly seem to be about regression models and deal with cross validation, comparison of nested statistical models via likelihood-ratio, etc. Apologies if these methods are applicable and I just don't realize how.
Problem: I want to compare two relatively complex simulation models which predict a time-series and test which one fits the observations better. No parameters are being fitted. These are not regression models.
I'd like to say whether or not the accuracy of the two models is significantly different, so a simple comparison of RMSEs isn't enough.
Example: Let's say I want to test which one of two numerical weather prediction models is better at predicting the temperature at place X (where I have a thermometer). Both models are mostly identical and rely on the same input variables, except for one process, which is modelled in a fundamentally different way (i.e. this is not just about adding a single parameter).
What kind of test would be suitable for this? I'm thinking about testing for differences in the residuals, but I'm not sure in what way exactly (i.e. do I test for differences in the residuals and just check which distribution is closer to zero? Do I compare absolute or squared residuals with a KS-Test? If yes, which one?). Or is there a simpler solution?
(Bonus: In my specific problem I have a strong suspicion which of the two models is more accurate, so a one-sided variant would be interesting.)