What measure should I use to compare two sets of calculations? Assume I have two sets of calculations produced by two different simulators.  There is no way to precisely measure actual values for these calculations, so I'm defaulting to the assumption that one of these simulators (the "incumbent") is the golden standard, and I'm tweaking the second simulator (the "contender") to produce results closer to the incumbent.
The question: how to define "closer"?
What I've done so far:


*

*Plotted each incumbent calculation on the x axis versus the contender calculation on the y axis.  For the majority of calculations, y is close to x, so most points fall on or near a line with a slope around 1.  However, with tens of thousands of calculations, the outliers dominate the scatterplot, making it hard to objectively say which of two such plots represents better agreement between the simulators.

*Computed RMSE between x and y.  This way I can tell if I've improved in one category, but as the units are different across categories, I can't say whether there is better agreement in one category versus another.

*Computed R-squared assuming a linear model of intercept 0 and slope 1.  I'm not sure this is an appropriate use of R-squared, which is normally calculated against the regression line.  It does however, allow for comparison across categories and provides an easy-to-understand metric for communication (the more they agree, the closer R-squared is to 1).

*Computed the regression line (lm(y~x)) and compared its slope to the ideal slope of 1.  This is a good way to see if, for example, one calculation is systematically off by a certain percentage, but it does nothing to communicate variability.


Which of these, or some other measure, is most appropriate for capturing the agreement between the two simulators, for the purpose of optimizing the performance of the contender to better match that of the incumbent?
 A: The RMSE should give a good indication of how well the two simulations match each other, and if you want to compare the results from different categories you could somehow normalize the RMSE value. A simple way to do this might be by dividing each RMSE by the mean of the given category. Wikipedia also has an entry on this. I have often used this to compare the results between categories. This method isn't perfect though because a very large or very small mean can lead to an abnormally large or small ratio.
If you want to see the residuals for individual points you can, of course, use a scatter plot with x and y being the results from each of the models. As you mentioned, these can be hard to read when there are thousands of points. In that case you could create a histogram with the bins being set to a certain range of residual values. If you wanted to compare the residuals for individual points across different categories, you could normalize the residuals by dividing each residual by the RMSE of its respective category.
Additionally, I often find it helpful to compare the RMSE to the accuracy of the reference data that I used to build the model, since a model cannot be more accurate than the reference data on which it is based. You mentioned though that the values produced by your simulator cannot be accurately measured. Out of curiousity, what sorts of values are you trying to predict?
