Is there a way to determine if there is a statistically significant difference between measurements obtained by two different methods?

Question

Here's my situation. I have 259 samples from a data set, and each sample has a value calculated by two different methods. Both methods are trying to describe the same phenomenon, and ideally when a measurement from method 1 is high, the measurement from method 2 should also be high, relative to the other samples. However, what I am interested in is samples in which this is not true. Is there a test I could run to find the samples in which there is a statistically significant difference between the two measurements (i.e. the samples in which the measurement with method 1 is high and the measurement with method 2 is low, or vice versa), or is this not an apples-to-apples comparison?

Thanks

Answer 1

So for m1 and m2, you could fit a linear model m1 ~ m2 and then look at the residuals for your data in the model. You could then either set some threshold of interest or measure the standard deviation of the residuals and pick (for example) observations that are more than 2 SD below the expected value.

Answer 2

There are many such statistical measurements of "interrater reliability" or "interrater agreement", as well as several good graphical methods like Jacob Weverka and mdewey mentioned.

One particularly useful statistic in the case of continuous data measured by two different methods (or individuals, or tools, ...) is the intraclass correlation, which you can see how to calculate on Wikipedia. The intraclass correlation will be close to 1 if the methods usually agree and close to 0 if they usually disagree. You can also calculate the standard correlation coefficients (like Pearson or Spearman), but usually for calculating agreement the ICC is a bit better (though the difference is not so important for only two raters).

Notably, this does not tell you how accurate either of the raters are, only how well they agree with each other (i.e. it does not really assess the measurement error) -- you would need a known standard to compare to if you also wanted to assess the accuracy.

Answer 3

Well, the first thing to do, like always, is to look at your data. The simplest way is just a scatter plot, $m1$ against $m2$. On the scatter plot, just add a good OLS old linear regression, and look at the Pearson correlation coefficient, and the slope and intercept of the OLS.
If the correlation is not even linear, you have some more work to do to investigate the 2 methods; they do not really seem to calculate the same thing, the same way...
If the linear correlation coefficient is low, your 2 methods mostly do not agree, and there is no point in digging any further.
Assuming a decent linear correlation, I would look at the slope and intercept; are they close to 1 and 0 respectively? If not, there is systematic bias bewteen the 2 methods; additive for a non-0 intercept, and multiplicative for non-1 slope. There is no point in comparing the 2 datasets until you correct for these potential systematic biases (think of this as "calibration"). Remove all the non-random, systematic biases, before you try to analyze the random errors.
Now you have correlated data, with no systematic biases. You can start comparing in more depth.
I would then determine whether I look at the differences between the 2 methods in absolute terms ($m_1-m_2$), or relative terms ($\frac {(m_1-m_2)} {m_1}$); this is domain dependent. E.g. if I measure glucose levels, I would use relative terms -a difference of 10 at a glucose level near 300mg/dl, or near 80mg/dl are clinically very different. But if I measure body temperature, the range is so small (80-110C?), that I can use absolute differences. For relative difference use either method (Bland-Altman uses the average of the 2).
Now, you can do a simple scatter plot, or a Bland-Altman plot (a variant of the scatter plot), using the "calibrated" datasets, and the (absolute or relative) differences.
The plots should already visually show you extreme values (aka outliers, but I do not like using this term). And you can compute the sd of the differences and use 1, 2, or 3 sd's as your criteria for flagging extreme values. But often, you would use another criteria; it may be that, in your specific domain, a difference of $x$, or $x$% is practically significant (e.g. 15% for glucose levels, 0.5C for temperature); therefore that is what you should use as your criteria, regardless of the sd.