Assume I have two sets of calculations produced by two different simulators. There is no way to precisely measure actual values for these calculations, so I'm defaulting to the assumption that one of these simulators (the "incumbent") is the golden standard, and I'm tweaking the second simulator (the "contender") to produce results closer to the incumbent.

The question: how to define "closer"?

What I've done so far:

  1. Plotted each incumbent calculation on the x axis versus the contender calculation on the y axis. For the majority of calculations, y is close to x, so most points fall on or near a line with a slope around 1. However, with tens of thousands of calculations, the outliers dominate the scatterplot, making it hard to objectively say which of two such plots represents better agreement between the simulators.
  2. Computed RMSE between x and y. This way I can tell if I've improved in one category, but as the units are different across categories, I can't say whether there is better agreement in one category versus another.
  3. Computed R-squared assuming a linear model of intercept 0 and slope 1. I'm not sure this is an appropriate use of R-squared, which is normally calculated against the regression line. It does however, allow for comparison across categories and provides an easy-to-understand metric for communication (the more they agree, the closer R-squared is to 1).
  4. Computed the regression line (lm(y~x)) and compared its slope to the ideal slope of 1. This is a good way to see if, for example, one calculation is systematically off by a certain percentage, but it does nothing to communicate variability.

Which of these, or some other measure, is most appropriate for capturing the agreement between the two simulators, for the purpose of optimizing the performance of the contender to better match that of the incumbent?

  • $\begingroup$ Are you trying to see which of 2 proxies better agrees with a gold standard, or just what the agreement is between a proxy & a gold standard? What kind of numbers are they (eg, continuous, bounded, categorical)? Do you want to try to calibrate the proxy, or predict the gold standard from the proxy? $\endgroup$ – gung - Reinstate Monica Jun 17 '16 at 16:58
  • $\begingroup$ Good questions. One proxy, one gold standard (or maybe two proxies and no gold standard, depending on how you look at it). Continuous variables, non-negative. Calibrating the proxy to the gold standard (or bringing the two proxies in line with each other). $\endgroup$ – teadotjay Jun 17 '16 at 18:18
  • $\begingroup$ It might help to point out the distributions are roughly normal with respect to the log of the calculation. So my scatterplots are on log scale. @gung I've been reading your posts in agreement-statistics and they have been helpful so far. $\endgroup$ – teadotjay Jun 17 '16 at 20:17

The RMSE should give a good indication of how well the two simulations match each other, and if you want to compare the results from different categories you could somehow normalize the RMSE value. A simple way to do this might be by dividing each RMSE by the mean of the given category. Wikipedia also has an entry on this. I have often used this to compare the results between categories. This method isn't perfect though because a very large or very small mean can lead to an abnormally large or small ratio.

If you want to see the residuals for individual points you can, of course, use a scatter plot with x and y being the results from each of the models. As you mentioned, these can be hard to read when there are thousands of points. In that case you could create a histogram with the bins being set to a certain range of residual values. If you wanted to compare the residuals for individual points across different categories, you could normalize the residuals by dividing each residual by the RMSE of its respective category.

Additionally, I often find it helpful to compare the RMSE to the accuracy of the reference data that I used to build the model, since a model cannot be more accurate than the reference data on which it is based. You mentioned though that the values produced by your simulator cannot be accurately measured. Out of curiousity, what sorts of values are you trying to predict?

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.