1
$\begingroup$

I'm trying to compare results predicted by a complex model with measured results and don't know what the best method would be to do so. RMSE could give you an idea of the models absolute accuracy, but it's more important to accurately predict the trend. Predicting the trend here does not only mean predicting whether it increases/decreases, but also that the slope should be estimated correctly.

I've looked into correlation coefficients (Pearson and Spearman), but those seem to be a bad fit. Pearson only tests for linear relationships (the model's predictions and measured results can, for example, follow a logarithmic trend). Spearman uses ranking solving the linear requirement but this leads to a problem because it doesn't actually tell you if the trend is estimated correctly, only if both the predictions and measured results increase/decrease monotically.

Example of why Spearman is bad:

input = 2, 4, 8

measured = 1.5, 3, 6

predicted = 1, 2, 3

Spearman(input,measured) = 1

Spearman(input,predicted) = 1

Comparing both correlation coefficients will, wrongly, tell you that your model is correct. In my case, the model is not correct enough because as the input variable doubles, it's not actually predicting that the output also doubles.

In the real model, there are many input variables and one output. The model can also be more sensitive to one input variable than to another.

The questions:

  • The inputs have different orders of magnitude but maybe I could normalize them (could this lead to wrong conclusion because of the different sensitivities)?

  • What would be the best method to correlate the results of prediction model to measured results?

  • Can I just compare two Pearson coefficients and say that, if they're alike, the model performs well (I don't think so because the data is not linear)?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.