Assume I have two sets of calculations produced by two different simulators. There is no way to precisely measure actual values for these calculations, so I'm defaulting to the assumption that one of these simulators (the "incumbent") is the golden standard, and I'm tweaking the second simulator (the "contender") to produce results closer to the incumbent.
The question: how to define "closer"?
What I've done so far:
- Plotted each incumbent calculation on the x axis versus the contender calculation on the y axis. For the majority of calculations, y is close to x, so most points fall on or near a line with a slope around 1. However, with tens of thousands of calculations, the outliers dominate the scatterplot, making it hard to objectively say which of two such plots represents better agreement between the simulators.
- Computed RMSE between x and y. This way I can tell if I've improved in one category, but as the units are different across categories, I can't say whether there is better agreement in one category versus another.
- Computed R-squared assuming a linear model of intercept 0 and slope 1. I'm not sure this is an appropriate use of R-squared, which is normally calculated against the regression line. It does however, allow for comparison across categories and provides an easy-to-understand metric for communication (the more they agree, the closer R-squared is to 1).
- Computed the regression line (lm(y~x)) and compared its slope to the ideal slope of 1. This is a good way to see if, for example, one calculation is systematically off by a certain percentage, but it does nothing to communicate variability.
Which of these, or some other measure, is most appropriate for capturing the agreement between the two simulators, for the purpose of optimizing the performance of the contender to better match that of the incumbent?