3
$\begingroup$

Using an example similar to the one from R predict, simulate some independent variable ($x$) data, map them to an observed dependent variable ($y$) using a linear function for the true value (I chose here $3 x - 1$), plus a random normal error. Then fit a linear model to the data.

set.seed(8298)
x <- runif(15, -3, 3)
x <- sort(x)
y <- 3 * x - 1 + rnorm(15)
lm1 <- lm(y ~ x)

Let $\hat y = f(x)$ be the linear model.

As explained by the R documentation on predict, $\hat y$ and its lower and upper limits can be calculated as:

new <- data.frame(x = seq(-3, 3, 0.5))
pred.w.plim <- predict(lm1, new, interval = "prediction")
pred.w.clim <- predict(lm1, new, interval = "confidence")

Then, if I want to represent the uncertainty on the predictions, I can plot $\hat y$ vs $x$, with lines that connect the upper limits of the intervals, and the same for the lower limits:

matplot(new$x, cbind(pred.w.clim, pred.w.plim[,-1]),
        lty = c(1,2,2,3,3), col = c(1,2,2,3,3), type = "l", lwd = 2,
        xlab = "x", ylab = "predicted y")

points(x, y, pch = 16)

enter image description here

Note that I added the experimental $y$ as points, to show how close the prediction (black line) is to the observed value (black point).

Now, imagine instead a case where $x$ is not a simple real variable, but a complicated vector of several descriptors, and the true relationship between $y$ and $x$ is no longer linear.
You can still make a model that outputs, for each object with a specific $x$, a prediction $\hat y$ with its upper and lower limits (of the 'prediction' type, in analogy with the linear case).
However, now you can no longer plot $\hat y$ vs $x$.
In such cases, we are told it is possible to plot $y$ (observed, experimental) vs $\hat y$:

y_pred <- predict(lm1)
plot(y ~ y_pred, pch = 16, xlab = "predicted y", ylab = "experimental y")
abline(0, 1)

enter image description here

In this case, to make an example, I will still calculate the prediction limits using the linear model, and with the 'confidence' method to show some variation between them, but you need to imagine that these limits come from some more complex calculation and they are actual prediction intervals.

old <- data.frame("x" = x)
pred.old.clim <- predict(lm1, old, interval = "confidence")

And this is where I am stuck, hence my post.
How do I correctly represent the uncertainty or interval on the predictions in such a plot?

Initially I hastily thought that I just needed to put vertical error bars on each point:

ints <- pred.old.clim[,"upr"] - pred.old.clim[,"lwr"]
segments(x0 = y_pred, y0 = y - ints / 2, y1 = y + ints / 2)

enter image description here

But this cannot be right, can it, as it uses an experimental value as the center for a distribution that should instead be centered on the predicted value.

The only two alternatives that seemed to make some sense were plotting the bars horizontally from each experimental point:

plot(y ~ y_pred, pch = 16, xlab = "predicted y", ylab = "experimental y")
abline(0, 1)
segments(y0 = y, x0 = pred.old.clim[,"lwr"], x1 = pred.old.clim[,"upr"])

enter image description here

or vertically, but from the unit line, i.e. from the point where prediction and experiment would agree:

plot(y ~ y_pred, pch = 16, xlab = "predicted y", ylab = "experimental y")
abline(0, 1)
segments(x0 = y_pred, y0 = pred.old.clim[,"lwr"], y1 = pred.old.clim[,"upr"])
matlines(x = pred.spr.clim[,1], y = pred.spr.clim[,-1], lty = 2, col = 2)
matlines(x = pred.spr.plim[,1], y = pred.spr.plim[,-1], lty = 3, col = 3)

enter image description here

What do you think is the correct way of doing this? Any of the above, something else...?

$\endgroup$

1 Answer 1

1
$\begingroup$

You are close to the usual approach in your second last plot. I would say that the convention is to place the observed (i.e. experimental) values on the X-axis and the fitted values on the Y-axis, assuming that are simply fitting a regression to experimental data. I think the term 'fitted' here is more clear because there is no implication that the values were predicted before the experiment was done. Then you can use vertical error bars to show the uncertainty in your fits. Note that it's also possible to have uncertainty in your observed values, in which case you can use both vertical and horizontal error bars (and you'd want to account for the uncertainty in X-values in the regression as well).

$\endgroup$
3
  • $\begingroup$ Thanks! So you would advise to represent the error bars centered on each experimental point, but swapping the axes. For some time we also used x = experimental, y = fitted, but then we were puzzled by the fact that the regression line was not close to the unit line, and we found this paper sciencedirect.com/science/article/abs/pii/… that proves that the 'correct' approach is actually x = fitted, y = experimental. So you see my conundrum, I want to do the right thing, but then these horizontal bars look weird... $\endgroup$ Commented Sep 15, 2022 at 18:39
  • $\begingroup$ As for the uncertainty on the observed values, indeed, that's true, although we normally do not have direct estimates of that (e.g. sample var from repeated measurements). The ML method we are using is in theory somehow able to separate the experimental variance contribution (aleatoric) from the model's own uncertainty (epistemic). See pubs.acs.org/doi/10.1021/acscentsci.1c00546 . From a practical point of view, however, the observed distances between predicted and observed are the sum of both contributions, and that is what I'd like to represent. $\endgroup$ Commented Sep 15, 2022 at 18:49
  • $\begingroup$ @user6376297 I'll have to take a closer look at that paper, thanks for pointing it out. I will say that (i) it's important to be clear about what is meant by predicted i.e. is it just a fit to the data or is it a prediction based on other data/theory/information? and (ii) orthogonal/Deming regression may be a solution to the the statistical concerns and may be more justified anyway. I'll need to read the paper and think about that more. $\endgroup$
    – mkt
    Commented Sep 15, 2022 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.