Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have two different analytical methods that can measure the concentration of a particular molecule in a matrix (for instance measure the amount of salt in water)
The two methods are different, and each has it's own error. What ways exist to show the two methods are equivalent (or not).
I'm thinking that plotting the results from a number of samples measured by both methods on a scatter graph is a good first step, but are there any good statistical methods ?
The simple correlation approach isn't the right way to analyze results from method comparison studies. There are (at least) two highly recommended books on this topic that I referenced at the end (1,2). Briefly stated, when comparing measurement methods we usually expect that (a) our conclusions should not depend on the particular sample used for the comparison, and (b) measurement error associated to the particular measurement instrument should be accounted for. This precludes any method based on correlations, and we shall turn our attention to variance components or mixed-effects models that allow to reflect the systematic effect of item (here, item stands for individual or sample on which data are collected), which results from (a).
In your case, you have single measurements collected using two different methods (I assume that none of them might be considered as a gold standard) and the very basic thing to do is to plot the differences ($X_1-X_2$) versus the means ($(X_1+X_2)/2$); this is called a bland-altman-plot. It will allow you to check if (1) the variations between the two set of measurements are constant and (2) the variance of the difference is constant across the range of observed values. Basically, this is just a 45° rotation of a simple scatterplot of $X_1$ vs. $X_2$, and its interpretation is close to a plot of fitted vs. residuals values used in linear regression. Then,
Other details may be found in (2), chapter 4.
References
If you have no way of knowing the true concentration, the simplest approach would be a correlation. A step beyond that might be to conduct a simple regression predicting the outcome on method 2 using method 1 (or vice versa). If the methods are identical the intercept should be 0; if the intercept is greater or less than 0 it would indicate the bias of one method relative to another. The unstandardized slope should be near 1 if the methods on average produce results that are identical (after controlling for an upward or downward bias in the intercept). The error in the unstandardized slope might serve as an index of the extent to which the two methods agree.
It seems to me that the difficulty with statistical methods here that you are seeking to affirm what is typically posed as a null hypothesis, that is, that there are no differences between the methods. This isn't a death blow for using statistical methods so long as you don't need a p value and you can quantify what you mean by "equivalent" and can decide how much deviation the two methods can have from one another before you no longer consider them equivalent. In the regression approach I detailed above, you might consider the methods equivalent if confidence interval around the slope estimate included 1 and the CI around the intercept included 0.
I agree with @drnexus. In addition, I might recommend a Morgan-Pitman test for the equality of variances of the two methods. This would tell you if one method has more variance than the other. This in itself might not be a bad thing because presumably the two tests have different bias-variance tradeoffs (for example, one test might always say 50% (biased, but no variance) while the other is unbiased but very noisy). Some domain knowledge might be helpful here in determining how much tradeoff you want of your method. Of course, as noted by others, having a 'gold standard' would be much preferred.
Quite an old question, but as it came up again today:
The general keyword is "validation in analytical chemistry" and as such it is a bit off-topic here (but as there is no Chemistry site here (yet: http://area51.stackexchange.com/proposals/4964/chemistry, I guess we can leave it here for the moment)
There are some standard procedures in analytical chemistry for this.
Books:
Funk et. al: Quality Assurance in Analytical Chemistry, Wiley-VCH.
Kromidas (Hrsg.): Handbuch Validierung in der Analytik, Wiley-VCH
(I don't know whether there is an English version and I don't have it (yet). But the table of contents lists validation of multivariate calibration.)
The IUPAC has something to say about that, too:
Danzer, K. and Currie, L. A.: Guidelines for calibration in analytical chemistry. Part I. Fundamentals and single component calibration, Pure and Applied Chemistry, IUPAC, 1998, 4, 993-1014
Danzer, K. and Otto, M. and Currie, L. A.: Guidelines for calibration in analytical chemistry. Part 2: Multicomponent calibration Pure and Applied Chemistry, 2004, 76, 1215-1225
Your use of the phrase 'analytical methods' is a bit confusing to me. I am going to assume that by 'analytical methods' you mean some specific model/estimation strategy.
Broadly, speaking there are two types of metrics you could use to choose between estimators.
In-sample Metrics
Out-of-sample Metrics
If the estimates are equivalent they would perform equally well on these metrics. You could also see if the estimates are not statistically different from one another (like the two-sample test of equality of means) but the methodology for this would depend on model and method specifics.
