5
$\begingroup$

My lab uses what our PI calls "modified Bland–Altman plots" to analyze regression quality. I'd like to know what the generally used name is for these plots.

A Bland–Altman plot compares the difference between two measures to their average. The "modification" is that the x-axis is, instead of the average, the ground truth value. The y-axis is still the difference between the predicted and true values. In effect, the modified B–A plot can be seen as the plot of residuals from the line $y=x$—i.e. the line $predicted=truth$.

B–A plots are also called mean–difference plots, so these modified ones could be considered "truth–difference" plots, but that name is not present in literature.

Admittedly, this example has terrible bias in its prediction, shown by the deviation from the zero-line.

modified bland altman plot

Returning to the question at hand: what is the accepted name for these plots?

$\endgroup$

2 Answers 2

8
$\begingroup$

On acceptance: Acceptance of names depends on who you want them to be accepted by. Bland-Altman plots are simply Tukey mean-difference plots (and Tukey was there much earlier), so if you want statisticians to accept the name you possibly wouldn't name it after Bland and Altman. On the other hand, in some application areas (medicine or chemistry, perhaps), you'd probably get quizzical looks if you called mean-difference plots anything but Bland-Altman. [However, chances are someone was there before Tukey as well, many of these ideas are quite old.]

On a suitable name: If those things you're calling "truth" are actually "truth" (not just observations with error, say), I'd probably just call what you have a residual plot (though it looks like those differences are negative residuals); depending on how your truth was obtained you might hyphenate in a descriptive noun after "residual" (e.g. if it was based on some gold-standard calibration you might call it a residual-standard plot).

If your truth is really truth (and not just observations or even some higher-quality estimate) then you could even argue that error should be used in place of residual.

On the suitability of the plot as a diagnostic: How were those "truth" values obtained? Are those just the actual data? If so "truth" is arguably a misnomer, and in that case you'd expect there to be some negative correlation in that plot; it wouldn't necessarily suggest a problem at all. We have many threads which explain (or even prove) that plots of $y-\hat{y}$ vs $y$ will have a positive correlation, if your "difference" axis is $\hat{y}-y$ then you'd expect a negative linear trend when the regression model was appropriate.

What is the aim of the plot? How are you interpreting it?


Here's a few of the existing posts relating to the issue of plotting residuals vs data:

Trend in residuals vs dependent - but not in residuals vs fitted

Does it make sense to study plots of residuals with respect to the dependent variable?

What is the expected correlation between residual and the dependent variable?

$\endgroup$
6
  • $\begingroup$ Thanks for all of this—the links suggest that I've been misinterpreting these plots, so another visualization would be superior. These "truths" are indeed observations; the scikit-learn package likes to call these y_true (as opposed to y_pred). Our hope is that there will be no correlation between predicted and "true" values. We're trying to find a good visual diagnostic of a regressor's quality. Plotting residuals against every feature is untenable, so the hope was to consolidate this into residuals against the true values. $\endgroup$ Commented May 23, 2017 at 21:52
  • 1
    $\begingroup$ Whoops—misspoke. Meant no correlation between residuals and "true" values—we want to be close to the true values always, without what looks like a bias in our estimate. $\endgroup$ Commented May 24, 2017 at 4:33
  • 1
    $\begingroup$ "Close to true" would mean small residuals. The usual plot of residuals vs fitted should be okay for that -- if you look at standardized residuals you'd combine it with the $s$ from regression to assess their actual typical size. $\endgroup$
    – Glen_b
    Commented May 24, 2017 at 4:34
  • 1
    $\begingroup$ Plotting difference versus mean can be seen in a paper by Neyman, Scott and Shane in 1953. Reference, examples and full context in Brillinger, David R. 2008. The 2005 Neyman Lecture: Dynamic Indeterminism in Science._ Statistical Science_ 23(1): 48-64. doi.org/10.1214/07-STS246 The late Douglas Altman was clear to me that he and Martin Bland were just using an old idea. I've seen examples from Bliss, Hills and Oldham independently of Tukey. As Stephen Stigler emphasised, naming any idea after individuals is likely to get ths history wrong. $\endgroup$
    – Nick Cox
    Commented Sep 12, 2023 at 7:03
  • 1
    $\begingroup$ I would advocate explanations of the form: we plotted !!! on the vertical axis versus ??? on the horizontal axis. Even versus isn't used by everyone in the same way. stats.stackexchange.com/questions/146533/… $\endgroup$
    – Nick Cox
    Commented Sep 12, 2023 at 7:08
2
$\begingroup$

Difference plots like your figure are indeed sometimes called "modified Bland-Altman plots". As you say, the term “modified” must be added because Bland and Altman recommended against using the comparator or reference measurement $C$ on the horizontal or “x” axis and instead said to use the average of the index measurement $M$ and the comparator measurement, $(M + C)/2$.

Assume that, on average, $M$ and $C$ differ by a constant, regardless of measurement level. A scatterplot of the difference $M-C$ vs. measurement level should be roughly flat. If you use $C$ instead of $(M+C)/2$ for the measurement level, you will get a spurious negative correlation between the difference and the measurement level. In other words, the regression line will have a negative slope even if the difference $M - C$ does not decrease or become more negative with magnitude.

Krouwer recommends using $C$ on the hoirxontal or "x" axis if $C$ is truly a precise reference value with small within-subject variance.

The plot that you showed looks like the index test always returned a measurement of $8$, so that, when the reference value was $5$, the difference was about $8-5=3$ and when the reference value was $30$, the difference was $8 - 30 = -22$. I'm guessing the $Corr(M-C,C) \approx -1$. A test that returns roughly the same number regardless of the true value is useless.

Derivation of negative correlation between $M-C$ and $C$:

$$M = b_M + X +e_{M}, \quad e_{M} \sim N(0,\sigma^2_M)$$ $$C = b_C + X +e_{C}, \quad e_{C} \sim N(0,\sigma^2_C)$$ $X = $ `true' value, a random variable from a distribution with variance $\sigma^2_X$

$M = $ index measurement (variance $\tau^2_M = \sigma^2_X + \sigma^2_M$)

$C = $ comparator measurement (variance $\tau^2_C = \sigma^2_X + \sigma^2_C$)

$b_M = $ systematic error for the index method

$b_C = $ systematic error for the comparator method

$Cov(M,C) = \sigma^2_X$

$Var(M-C) = \sigma^2_M + \sigma^2_C$ \begin{align*} Cov(M-C,C) &= Cov(M,C) - \tau^2_C\\ &= \sigma^2_X - (\sigma^2_X + \sigma^2_C)\\ & = -\sigma^2_C\\ Corr(M-C, C) &= \frac{-\sigma^2_C}{\sqrt{\sigma^2_M + \sigma^2_C}\sqrt{ \sigma^2_X + \sigma^2_C}}\\ \end{align*} So the correlation coefficient between M-C and C should be negative.

Bland and Altman do not consider the true value $X$, but they do use the correlation coefficient $\rho$ between $M$ and $C$. \begin{align*} Cov(M-C, C) &= \rho \tau_M \tau_C - \tau^2_C\\ Corr(M-C,C) &= \frac{\rho \tau_M \tau_C - \tau^2_C}{\sqrt{ \tau^2_M + \tau^2_C - 2\rho\tau_M\tau_C}\cdot (\tau_C)}\\ &= \frac{\rho \tau_M - \tau_C}{\sqrt{ \tau^2_M + \tau^2_C - 2\rho\tau_M\tau_C}}\ \end{align*} If $\tau_M = \tau_C$, then $$Corr(M-C, C) = -\sqrt{\frac{1-\rho}{2}}$$ A typical value for $\rho$ is 0.98, so a typical value for the correlation is -0.1. In the absence of a genuine association between the difference $M-C$ and measurement magnitude, the slope of a regression line fit to $M - C$ vs. $C$ will still be negative.

$\endgroup$
6
  • $\begingroup$ "Assume that, on average, M and C differ by a constant" This sentence is a bit unclear. How does it look like to differ on average by a constant? It could mean that the conditional mean of M-C is constant as a function of some other value. But what value is meant? If it is constant as function of C, then you get the Krouwer situation. If it is constant as function of (C+M)/2, then you get the Bland and Altman situation. $\endgroup$ Commented Sep 11, 2023 at 6:33
  • $\begingroup$ I mean $M_i - C_i \sim N(b_M - b_C, \sigma_M^2 + \sigma_C^2)$. If $b_M = -10$ and $b_C = 0$, then, on average, $M - C =-10$. The points in the scatterplot should surround a horizontal line at difference = -10. But if $\sigma_M \ne \sigma_C$, then plotting $M-C$ vs $C$, won't produce a horizontal line; it will have a negative slope. $\endgroup$
    – Mkanders
    Commented Sep 11, 2023 at 23:45
  • $\begingroup$ @Mkanderd I get the idea and can fill in what is meant. It is just that the phrase is a bit weird because you speak about 'differ' and 'constant' in the same sentence. That makes it ambiguous. $\endgroup$ Commented Sep 12, 2023 at 4:52
  • $\begingroup$ Your last comment is not entirely correct. It only states the marginal distribution of M-C. If I generate it in R as follows C = rnorm(100,0,1); M = C + rnorm(100,1,1) then M-C follows that normal distribution, but there is no correlation between M-C and C. $\endgroup$ Commented Sep 12, 2023 at 4:55
  • $\begingroup$ I added a derivation of the negative correlation between $M-C$ and $C$. This is based on the Bland-Altman paper that I added as a reference. I welcome corrections to this derivation and have considered posting it as a question. $\endgroup$
    – Mkanders
    Commented Sep 12, 2023 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.