How to compare observed and expected outcomes for continuous data

Question

I am working on some data, more specifically some predictions of some outcomes. The predictions vary on the continuous scale, between $-3$ and $3$. They can for example be:

$x_1=-2.4, x_2=-2.1, x_3=1.4, x_4=0.4, \cdots, x_n=-1.2$

Now, for each prediction, I also have the corresponding results,

$y_1, y_2, \cdots , y_n$,

which also are continuous.

How can I check if my predictions are good or not? I do not have any knowledge whatsoever on how the predictions are made, thus cannot create any confidence interval around the predictions.

Answer 1

In the first instance you should just look at a scatter plot of observed and predicted and think about e.g. the correlation between the two.

However, there is a simple but important twist: the reference or ideal situation here is agreement, i.e. $y = x$, not the more general linear relationship $y = a + bx$. So, correlation can't quite be the answer to your question. For example, $y = x + 100$ and $y = 10x$ would both give correlations of 1 but additive or multiplicative bias in your predictions would presumably not be acceptable.

Concordance correlation is moderately well-known as a way to collapse comparisons to a single number. See e.g. Does the concordance correlation coefficient make linearity or monotone assumptions? and its references.

Another way to think is simply that the difference between observed and predicted is a residual from your unstated model; then use as many of the ways of thinking about residuals as you can apply: see any good regression text for residual plotting ideas. Often the fine structure revealed by such plots is as useful as concordance correlation, which quantifies strength of agreement without revealing systematic patterns in disagreement between observed and predicted.

Not knowing how the predictions were generated is unfortunate, but presumably beyond your control.