# What ways are there to show two analytical methods are equivalent?

I have two different analytical methods that can measure the concentration of a particular molecule in a matrix (for instance measure the amount of salt in water)

The two methods are different, and each has it's own error. What ways exist to show the two methods are equivalent (or not).

I'm thinking that plotting the results from a number of samples measured by both methods on a scatter graph is a good first step, but are there any good statistical methods ?

• Can you give more details in your question? I don't understand what is "the concentration of a particular molecule in a matrix". Jul 24, 2010 at 6:41
• @robin: "matrix" in this context is standard analytical chemistry terminology; it refers to the medium where the entities to be analyzed for (the "analytes") can be found. For instance, if you're analyzing for the concentration of lead in tap water, lead is the analyte, and water is the matrix. Sep 18, 2010 at 0:49

The simple correlation approach isn't the right way to analyze results from method comparison studies. There are (at least) two highly recommended books on this topic that I referenced at the end (1,2). Briefly stated, when comparing measurement methods we usually expect that (a) our conclusions should not depend on the particular sample used for the comparison, and (b) measurement error associated to the particular measurement instrument should be accounted for. This precludes any method based on correlations, and we shall turn our attention to variance components or mixed-effects models that allow to reflect the systematic effect of item (here, item stands for individual or sample on which data are collected), which results from (a).

In your case, you have single measurements collected using two different methods (I assume that none of them might be considered as a gold standard) and the very basic thing to do is to plot the differences ($X_1-X_2$) versus the means ($(X_1+X_2)/2$); this is called a . It will allow you to check if (1) the variations between the two set of measurements are constant and (2) the variance of the difference is constant across the range of observed values. Basically, this is just a 45° rotation of a simple scatterplot of $X_1$ vs. $X_2$, and its interpretation is close to a plot of fitted vs. residuals values used in linear regression. Then,

• if the difference is constant (constant bias), you can compute the limit of agreement (see (3))
• if the difference is not constant across the range of measurement, you can fit a linear regression model between the two methods (choose the one you want as predictor)
• if the variance of the differences is not constant, try to find a suitable transformation that makes the relationship linear with constant variance

Other details may be found in (2), chapter 4.

References

1. Dunn, G (2004). Design and Analysis of Reliability Studies. Arnold. See the review in the International Journal of Epidemiology.
2. Carstensen, B (2010). Comparing clinical measurement methods. Wiley. See the companion website, including R code.
3. The original article from Bland and Altman, Statistical methods for assessing agreement between two methods of clinical measurement.
4. Carstensen, B (2004). Comparing and predicting between several methods of measurement. Biostatistics, 5(3), 399–413.
• would you mind clarifying what you mean by "(a) our conclusions should not depend on the particular sample used for the comparison"? I'm having trouble due to the ambiguity of "sample" in this context: does it mean "statistical sample" (a set of data presumed to represent a process or population) or "environmental sample" (a bit of water, soil, air, or tissue, typically). With either meaning I can't quite draw the logical line to your conclusion that this "precludes any method based on correlations."
– whuber
Sep 19, 2010 at 18:36
• @whuber Well, I mean the collection of observed data (e.g. glucose concentration) which, ideally, should be representative of the likely range of what is being measured. Relying on correlation may be misleading because it depends on the sampled units (e.g. patients in an hospital): we can get a higher correlation just by getting one or more extreme measurement on either scale, although the relation between the two methods is still the same. Hence, the idea is that the distribution of the measure of interest should not influence our conclusion about methods comparability. (...)
– chl
Sep 19, 2010 at 19:16
• @whuber (...) What we want to assess is the agreement beyond the data, not the relationship in the data (I'm quoting Carstensen 2010 p. 8-9).
– chl
Sep 19, 2010 at 19:17
• Thank you; that clarifies your position well. This is essentially an exercise in calibration except that we do not appear to have a reference standard for comparison; we merely assume that the physical samples chosen by the experimenter cover some range of true concentrations. Thus, as you write, correlation per se is not necessarily a useful measure of agreement among the two methods. Typically though, especially for chemical analyses, the true concentration is known (because the experimenter introduced a known amount of a substance into the matrix).
– whuber
Sep 20, 2010 at 14:25
• @whuber That's right. In the absence of a gold standard, we are merely interested in the extent to which the two methods yield "comparable" results, hence the idea of relying on so-called limits of agreement. Although the true measure may be known in advance, each measurement instrument carry its own measurement error -- at least for those I used to deal with in the biomedical (e.g. blood glucose concentration) and neuropsychological (e.g. depression level) domain.
– chl
Sep 20, 2010 at 14:42

If you have no way of knowing the true concentration, the simplest approach would be a correlation. A step beyond that might be to conduct a simple regression predicting the outcome on method 2 using method 1 (or vice versa). If the methods are identical the intercept should be 0; if the intercept is greater or less than 0 it would indicate the bias of one method relative to another. The unstandardized slope should be near 1 if the methods on average produce results that are identical (after controlling for an upward or downward bias in the intercept). The error in the unstandardized slope might serve as an index of the extent to which the two methods agree.

It seems to me that the difficulty with statistical methods here that you are seeking to affirm what is typically posed as a null hypothesis, that is, that there are no differences between the methods. This isn't a death blow for using statistical methods so long as you don't need a p value and you can quantify what you mean by "equivalent" and can decide how much deviation the two methods can have from one another before you no longer consider them equivalent. In the regression approach I detailed above, you might consider the methods equivalent if confidence interval around the slope estimate included 1 and the CI around the intercept included 0.

• In chemometrics the instrument responses are often nonlinear and heteroscedastic. At a minimum that imposes a certain amount of caution when conducting and interpreting the regression.
– whuber
Sep 20, 2010 at 19:06

I agree with @drnexus. In addition, I might recommend a Morgan-Pitman test for the equality of variances of the two methods. This would tell you if one method has more variance than the other. This in itself might not be a bad thing because presumably the two tests have different bias-variance tradeoffs (for example, one test might always say 50% (biased, but no variance) while the other is unbiased but very noisy). Some domain knowledge might be helpful here in determining how much tradeoff you want of your method. Of course, as noted by others, having a 'gold standard' would be much preferred.

Quite an old question, but as it came up again today:

The general keyword is "validation in analytical chemistry" and as such it is a bit off-topic here (but as there is no Chemistry site here (yet: http://area51.stackexchange.com/proposals/4964/chemistry, I guess we can leave it here for the moment)

There are some standard procedures in analytical chemistry for this.

Books:

• Funk et. al: Quality Assurance in Analytical Chemistry, Wiley-VCH.

• Kromidas (Hrsg.): Handbuch Validierung in der Analytik, Wiley-VCH
(I don't know whether there is an English version and I don't have it (yet). But the table of contents lists validation of multivariate calibration.)

The IUPAC has something to say about that, too:

• Danzer, K. and Currie, L. A.: Guidelines for calibration in analytical chemistry. Part I. Fundamentals and single component calibration, Pure and Applied Chemistry, IUPAC, 1998, 4, 993-1014

• Danzer, K. and Otto, M. and Currie, L. A.: Guidelines for calibration in analytical chemistry. Part 2: Multicomponent calibration Pure and Applied Chemistry, 2004, 76, 1215-1225

Your use of the phrase 'analytical methods' is a bit confusing to me. I am going to assume that by 'analytical methods' you mean some specific model/estimation strategy.

Broadly, speaking there are two types of metrics you could use to choose between estimators.

In-sample Metrics

• Likelihood ratio / Wald test / Score test
• R2
• In-sample Hit Rates (Percentage of correct predictions for sample data)
• (Lots of other metrics depending on model / estimation context)

Out-of-sample Metrics

• Out-of-sample Hit Rates (Percentage of correct predictions for out-of-sample data)

If the estimates are equivalent they would perform equally well on these metrics. You could also see if the estimates are not statistically different from one another (like the two-sample test of equality of means) but the methodology for this would depend on model and method specifics.

• Sorry, I was meaning an analytical measurement method. I've re-worded the question. Jul 23, 2010 at 20:58
• In that case, I think the two-sample test of equality for means/proportions is what you may want to do.
– user28
Jul 23, 2010 at 21:04
• Wouldn't a test of means/proportions only give you a point estimate of whether the two methods gave the same average response for a given set of responses? Couldn't that approach yield a result of "equal" even if the two methods were actually negatively correlated with one another? Jul 24, 2010 at 7:44
• That is a good point.
– user28
Jul 24, 2010 at 12:16