1
$\begingroup$

I'm predicting a response measured by a 5-point likert scale from a model that produces continuous predictions of the same variable. For example:

prediction:    outcome:
3.221          3
3.623          4
1.231          1
...            ...

How predictive accuracy should be measured in this case?

Mean Square Error (MSE) seems to be somewhat unfair because the model can never attain zero MSE - the quantization of the outcome introduces error that isn't related to the model's predictive accuracy (as in the above example).

Ordinal correlation measures (e.g. Spearman's R or Kendall's Tau) seem to throw too much data away: For example, if the ground truth (real outcome) is [3,4,1], they would assign the same accuracy to a model that predicts [3,4,1] and to a model that predicts [2.1,2.2,2.099].

Converting the model's continuous predictions to ordinal predictions by ordinal regression or by isotonic regression seem to suffer from the same problem (throwing away magnitude information).

Edit: Simple rounding can form predictions that are fully compatible with the outcome measure. Yet this seems to be suboptimal since it accounts for quantization error in one variable by introducing quantization error in another variable. I'd prefer an accuracy/error measure that views the outcome as reflecting a latent continuous variable.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ It seems to me you are missing a step: you don't actually have a prediction. You need to specify a procedure that unambiguously translates each numerical value currently labeled "prediction" into a unique possible outcome value. Once you have accomplished that, the meaning of "prediction error" will be clear and what remains is to define a loss function to translate errors into some cost that (a) is meaningful to your problem and (b) makes sense to aggregate across many individuals. Please edit your post to indicate which of these two issues you are concerned with. $\endgroup$
    – whuber ♦
    Commented Jun 13, 2019 at 19:11
  • $\begingroup$ Thank you @whuber, the way I see it, the continuous predictions I have are 'real' - the problem is with the quantized outcome. Obviously, I can transform the continuous predictions into ordinal predictions that will perfectly match the outcome, but this throws relevant information about the predictions' quality (predicting 3 by 3.1 is better than predicting 3 by 3.49, you can't see that if you rounded the predictions). $\endgroup$
    – Trisoloriansunscreen
    Commented Jun 13, 2019 at 20:43
  • $\begingroup$ That begs the question of how these predictions will be used: in practice, how would one react to a prediction of, say, 3.4? $\endgroup$
    – whuber ♦
    Commented Jun 13, 2019 at 21:21
  • $\begingroup$ @whuber, the context is scientific, not applied, so the predictions serve only the testing of the underlying model, and more importantly, comparing it to alternative models. Perhaps a way to paraphrase this problem is to consider the evaluation of regression models in a scenario in which our outcome measurements were recorded with a very limited precision. $\endgroup$
    – Trisoloriansunscreen
    Commented Jun 14, 2019 at 4:48
  • $\begingroup$ I see no distinction between "scientific" and "applied:" by its very nature, any use of mathematics in science is an application of mathematics. Comparing your predictions to those of alternative models is quite different than comparing your predictions to the ordinal values! $\endgroup$
    – whuber ♦
    Commented Jun 14, 2019 at 12:43

1 Answer 1

Reset to default
-1
$\begingroup$

Don't you have confusion matrix in the software you use for that? Correct hits can be easily calculated as % of total.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.