9
$\begingroup$

I find that Bland-Altman plots for comparing two methods are extremely useful in assessing agreement. However, I'm curious if there is a similar method or transformation that can be used when the scales of the two methods are not identical, but they still measure the same underlying phenomenon.

For example, I'm attempting to compare the agreement between two methods that both measure protein content: quantitative Western blot and tryptophan fluorescence. Each gives a very different kind of measurement, but it's still reasonable to question how well they agree in measuring protein content.

My question: Is there a method similar to a Bland-Altman plot (also known as a Tukey Mean-Difference plot) that can handle different scales? My only thought was to use a Spearman correlation to compare the relative ordering of samples, but correlations are fraught with peril. Many thanks for any responses.

$\endgroup$
3
  • $\begingroup$ I'm not sure it makes any sense to do this. Part of the purpose is to compare the means to look for bias, if you're on different scales the means aren't meaningful. $\endgroup$ Aug 14, 2015 at 17:42
  • $\begingroup$ Thank you @Jeremy for your thoughts. I'm happy to broaden the question then. I'm seeking a way to compare the agreement of two orthogonal methods that assess the same underlying phenomenon (like protein content) when numerically those methods are not on the same scale. Spearman correlation to compare rank was my best thought, but that feels like an underwhelming comparison. $\endgroup$
    – Ashe
    Aug 18, 2015 at 14:41
  • 1
    $\begingroup$ A very similar, possibly duplicate, question: stats.stackexchange.com/questions/114465/… $\endgroup$
    – Ashe
    Aug 19, 2015 at 12:44

3 Answers 3

3
$\begingroup$

The problem with using correlations as a measure of agreement is that what they are really assessing is the ordering of the $X_i$ and $Y_i$ values, and their relative spacing, but not that the numbers themselves agree (cf., see my answer here: Does Spearman's $r=0.38$ indicate agreement?). On the other hand, if the numbers are incommensurate, it makes no sense to try to determine if they agree—it can't mean anything whether they do or don't. As a result, a Bland-Altman plot can't be of any value here. However, a correlation might offer some (albeit little) value.

From an exploratory point of view, I would start with a regular, old scatterplot. I might also do a simple linear regression and test for curvature in the relationship. It can often be the case that different measures are differentially sensitive at different ranges. For example, they might do equally well at measuring what you want in the middle of their range, but one does a better job of measuring lower values (whereas the other just starts to output the same low number, perhaps a limit of detection), and vice-versa for higher values. What I have in mind is that that the relationships aren't linear. Consider this stylized figure of the relationship between energy and the temperature of water:

enter image description here

Then imagine having temperature and something else, perhaps volume (ice begins to expand at lower temperatures), both as measures of energy.

Once / if you were satisfied that the relationship were linear, your ability to measure the amount of agreement would be limited to Pearson's product-moment correlation; Bland-Altman plots just won't work here.

$\endgroup$
1
  • 1
    $\begingroup$ After a year of noodling on this, I think this approach would be the best in my situation. Start with a visual assessment, then consider how the ordering/spacing compare, and check for differences in sensitivities via regression. Thank you for giving it some thought. $\endgroup$
    – Ashe
    Dec 2, 2016 at 15:44
1
$\begingroup$

Assuming you cannot convert both measures to a common set of units, and both measures are continuous and are roughly normally distributed, convert both to standardized scores (e.g., $z = \frac{x- \mu}{\sigma}$).


Added in response to @Nick: Bland-Altman plots plot the difference between two measures against the average of the two measures, so to be meaningful the two measures need to be measured on the same scale. Converting two measures that are on different scales to dimensionless standardised scores allows you to do the necessary calculations.


Added in response to @Nick (2):

Not sure what you are saying. Here is a workable example:

# Load packages
library(dplyr)
library(BlandAltmanLeh)

# Using the same conditions @Ashe used

## Set seed
set.seed(2063)

## Generate data
x <- seq(1, 40)
y <- 2 * x + rnorm(n = length(x), mean = 0, sd = 10)

## Put x and y into a dataframe
df <- data_frame(x = x,
                 y = y) %>%
    ## Add two new columns containing standarized values of x and y
    mutate(x_std = (x - mean(x)) / sd(x),
           y_std = (y - mean(y)) / sd(y))

## Bland-Altman plots of:
### i) raw x and y values
raw <- bland.altman.plot(group1 = df$x,
                  group2 = df$y,
                  main = 'Raw values',
                  xlab = 'Average of x and y',
                  ylab = 'Difference between x and y')

enter image description here

### ii) standardized x and y values
std <- bland.altman.plot(group1 = df$x_std,
                  group2 = df$y_std,
                  main = 'Standardized values',
                  xlab = 'Average of x and y',
                  ylab = 'Difference between x and y')

enter image description here

It achieves the same (in shape at least) result as the lm approach used by @Ashe, which is what you would expect since both methods are 'rescaling' the values.

$\endgroup$
5
  • 1
    $\begingroup$ Why would this help? I think this answer needs expansion. $\endgroup$
    – Nick Cox
    Jul 2, 2016 at 17:19
  • 1
    $\begingroup$ Sure, but what would you plot and why does it help? Standardizing both variables just changes the units on a scatter plot. Plotting differences and means for standardized variables throws out the differences in location and scale that need to be quantified, i.e. it's a shift and a tilt that are most interesting and important here, and you've thrown them both out. $\endgroup$
    – Nick Cox
    Jul 2, 2016 at 18:13
  • $\begingroup$ Now the scatter is necessarily around a mean difference of zero. I don't see how that helps illuminate the real structure; you threw out the baby with the bathwater. The graph says less about the real data than a plain difference versus mean plot. $\endgroup$
    – Nick Cox
    Jul 4, 2016 at 8:29
  • $\begingroup$ Aaha, it took a while for the penny to drop over what your issue with standardization was, and I agree that standardization removes the 'colour' from the data. The main issue standardization solved was being able to do the maths (mean and differences between the two measures), but PCA (as you suggested) is a much better solution. $\endgroup$
    – Peter K
    Jul 4, 2016 at 18:23
  • $\begingroup$ Thank you for your contribution, Peter K. The message I'm getting is that, ultimately, using the Bland-Altman methodology isn't useful when the scales are different. Something @JeremyMiles said in a comment in the beginning. So the answer to my original question is simply, "No." $\endgroup$
    – Ashe
    Jul 5, 2016 at 19:22
0
$\begingroup$

I came up with a possible solution, so I will attempt to answer my own question. I would like some critical feedback from the community though.

I know the two phenomenon are related, so I make the assumption that I can calibrate one scale to the other scale. I will then compare agreement between the predicted values from one method to the experimental values of the other method. This method still cannot find bias in means (as @Jeremy pointed out, this isn't meaningful in this context), but it still might allow a comparison of the 95% limits.

Some code (in R) to compare:

library(ggplot2)
set.seed(2063)  #Dr. Cochrane

bland <- function(x, y, titl=''){
  gg.data <- data.frame(x=x, y=y, avg=(x+y)/2, diff=(x-y))
  g <- ggplot(gg.data, aes(x=avg, y=diff)) + geom_point(size=4) + theme_bw()
  g <- g + theme(text=element_text(size=24), axis.text=element_text(colour='black'))
  g <- g + labs(x='Average', y='Difference') + ggtitle(titl)
  g <- g + geom_hline(yintercept=mean(gg.data$diff), colour='chocolate', size=1)
  g <- g + geom_hline(yintercept=mean(gg.data$diff) + 1.96*sd(gg.data$diff), colour='dodgerblue3', size=1,
                      linetype='dashed')
  g <- g + geom_hline(yintercept=mean(gg.data$diff) - 1.96*sd(gg.data$diff), colour='dodgerblue3', size=1,
                      linetype='dashed')
  plot(g)

}

#Make some data
x <- seq(1,40)
y <- 2*x + rnorm(n=length(x), mean=0, sd=10)

qplot(x,y)
lm.data <- data.frame(x=x, y=y)

lm(data=lm.data, y~x)

#Bland-Altman of raw data
bland(x,y,'Raw Data')

#Bland-Altman of calibrated data
orig.df <- data.frame(x=x)
y.p <- predict(lm(data=lm.data, y~x), newdata=orig.df)

bland(y.p,y, 'Calib Data')
qplot(y.p,y)

If I try to directly compare $x$ and $y$, as expected, I get what would be very poor agreement: enter image description here

However, if I "calibrate" the $x$ values to the $y$ scale using a linear model, agreement appears much better: enter image description here

Some key thoughts:

  1. I don't have to use a linear model. Any model that calibrates one scale to another would do nicely.
  2. This is functionally equivalent to plotting the model residuals against the mean of $y$ and the $\hat{y}$ value. This is my biggest concern. I want to compare agreement between methods, but I could be simply evaluating the quality of the model. My current thinking is that these two are equivalent.
  3. Given #2, by comparing the residuals of the model as a measure of agreement, the value of my comparison rests strongly on the assumption that the model used to calibrate is correct.

To bring it all together, if I have selected a reasonable model (#1) to calibrate one scale to another (#3), then I can reasonably compare the residuals of that model (#2) as a measure of agreement. In the 2nd example graph above, I would interpret this as 95% of all deviations are within ~20 points on the $y$ scale. I can then evaluate if these limits are reasonable for the two methods I'm trying to study.

As I said above, criticisms are welcome.

$\endgroup$
6
  • 1
    $\begingroup$ I am agnostic on the question as Bland-Altman-Tukey plots (which were no doubt invented by someone else, possibly Thiele) are here predicated on agreement (meaning identity) as the reference situation. If your reference is linearity and wish to treat variables symmetrically then the two principal components are, I suggest, a suitable reordering of the data. The first PC measures agreement in that sense and the second PC measures disagreement and a plot of the two PCs is then appropriate. $\endgroup$
    – Nick Cox
    Feb 15, 2016 at 17:00
  • $\begingroup$ But further any model that calibrates one scale to another is a blank cheque (check) I can't endorse for you. If any one-to-one relationship is allowed, then OK, but you must first fit it, then plot residuals versus fit. $\endgroup$
    – Nick Cox
    Feb 15, 2016 at 17:03
  • $\begingroup$ @NickCox Thank you for the response. I like the suggestion of looking at the principal components. I operate in a situation where my group develops new molecular tests, and we compare them to reference standards. Ultimately, I want to be able to say with assurance that a new test matches established methodology. This gets murky when the two methods are on different scales. I was attempting to substitute linearity for agreement, and I like your approach with PCA more. $\endgroup$
    – Ashe
    Feb 15, 2016 at 17:53
  • $\begingroup$ @NickCox Can you elaborate, though, on what you mean by "one-to-one relationship"? Do you mean that there is 1 reference method for each new test? If that is the case, then no, there is not a one-to-one relationship. $\endgroup$
    – Ashe
    Feb 15, 2016 at 17:53
  • $\begingroup$ I mean a one-to-one mapping or monotonic relationship between your two scales, which I guess is precisely what you meant by any model that calibrates one scale to another. $\endgroup$
    – Nick Cox
    Feb 15, 2016 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.