2
$\begingroup$

I am working on some data, more specifically some predictions of some outcomes. The predictions vary on the continuous scale, between $-3$ and $3$. They can for example be:

$x_1=-2.4, x_2=-2.1, x_3=1.4, x_4=0.4, \cdots, x_n=-1.2$

Now, for each prediction, I also have the corresponding results,

$y_1, y_2, \cdots , y_n$,

which also are continuous.

How can I check if my predictions are good or not? I do not have any knowledge whatsoever on how the predictions are made, thus cannot create any confidence interval around the predictions.

$\endgroup$

1 Answer 1

2
$\begingroup$

In the first instance you should just look at a scatter plot of observed and predicted and think about e.g. the correlation between the two.

However, there is a simple but important twist: the reference or ideal situation here is agreement, i.e. $y = x$, not the more general linear relationship $y = a + bx$. So, correlation can't quite be the answer to your question. For example, $y = x + 100$ and $y = 10x$ would both give correlations of 1 but additive or multiplicative bias in your predictions would presumably not be acceptable.

Concordance correlation is moderately well-known as a way to collapse comparisons to a single number. See e.g. Does the concordance correlation coefficient make linearity or monotone assumptions? and its references.

Another way to think is simply that the difference between observed and predicted is a residual from your unstated model; then use as many of the ways of thinking about residuals as you can apply: see any good regression text for residual plotting ideas. Often the fine structure revealed by such plots is as useful as concordance correlation, which quantifies strength of agreement without revealing systematic patterns in disagreement between observed and predicted.

Not knowing how the predictions were generated is unfortunate, but presumably beyond your control.

$\endgroup$
2
  • $\begingroup$ Thank you for your reply! I am not sure what you mean by "...but additive or multiplicative bias in your predictions would presumably not be acceptable". I shall read about concordance and look at the 'residual plots'. Thank you very much, again! $\endgroup$
    – Erosennin
    Mar 3, 2014 at 11:28
  • 1
    $\begingroup$ By additive bias I mean predicted = observed $+$ constant and by multiplicative bias I mean predicted = observed $\times$ constant. $\endgroup$
    – Nick Cox
    Mar 3, 2014 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.