# What measure should I use to compare two sets of calculations?

Assume I have two sets of calculations produced by two different simulators. There is no way to precisely measure actual values for these calculations, so I'm defaulting to the assumption that one of these simulators (the "incumbent") is the golden standard, and I'm tweaking the second simulator (the "contender") to produce results closer to the incumbent.

The question: how to define "closer"?

What I've done so far:

1. Plotted each incumbent calculation on the x axis versus the contender calculation on the y axis. For the majority of calculations, y is close to x, so most points fall on or near a line with a slope around 1. However, with tens of thousands of calculations, the outliers dominate the scatterplot, making it hard to objectively say which of two such plots represents better agreement between the simulators.
2. Computed RMSE between x and y. This way I can tell if I've improved in one category, but as the units are different across categories, I can't say whether there is better agreement in one category versus another.
3. Computed R-squared assuming a linear model of intercept 0 and slope 1. I'm not sure this is an appropriate use of R-squared, which is normally calculated against the regression line. It does however, allow for comparison across categories and provides an easy-to-understand metric for communication (the more they agree, the closer R-squared is to 1).
4. Computed the regression line (lm(y~x)) and compared its slope to the ideal slope of 1. This is a good way to see if, for example, one calculation is systematically off by a certain percentage, but it does nothing to communicate variability.

Which of these, or some other measure, is most appropriate for capturing the agreement between the two simulators, for the purpose of optimizing the performance of the contender to better match that of the incumbent?

• Are you trying to see which of 2 proxies better agrees with a gold standard, or just what the agreement is between a proxy & a gold standard? What kind of numbers are they (eg, continuous, bounded, categorical)? Do you want to try to calibrate the proxy, or predict the gold standard from the proxy? Commented Jun 17, 2016 at 16:58
• Good questions. One proxy, one gold standard (or maybe two proxies and no gold standard, depending on how you look at it). Continuous variables, non-negative. Calibrating the proxy to the gold standard (or bringing the two proxies in line with each other). Commented Jun 17, 2016 at 18:18
• It might help to point out the distributions are roughly normal with respect to the log of the calculation. So my scatterplots are on log scale. @gung I've been reading your posts in agreement-statistics and they have been helpful so far. Commented Jun 17, 2016 at 20:17