0
$\begingroup$

I have two models/indices that try to predict observed values. I've compared them using correlation and regression, but I'd like to use MAE (Mean absolute error) to asses which of them is more closer to the observed values. The problem I ran into, is that they use different scales.

The observed values range from 0-100, the first model has the range of 0-100 as well, but the second one produces results that range from -1 to 1. This is due to the first model actually trying to predict the phenomenon collected in the observed values. The second model is trying to predict another phenomenon, but I'm trying to test whether it is also suitable for predicting the observed phenomenon. They both use different data and it would be difficult to standardize the data before running the model, because they so methodologically different.

I tried normalizing the observed and predicted values into z-scores with z = (x - mean(x))/ stdev(x) where x is a raw value, mean(x) is the mean of x population and stdev(x) the stdev of x population

Then calculated MAE with sum(abs(obs_z-pred_z)/count(obs) based on the z-scores. I got some values from this, but I'm not sure whether this a good method.

I also tried testing SMAE (Standardized mean absolute error) with both mean(abs(obs-pred)) / mean(obs) and mean(abs(obs-pred)) / stdev(obs) and these produce relatively similar results to eachother, but completely different to the z-scored MAE.

So therefore, is there a good or established method for comparing errors of models that produce values different scales? Ideally I'll have more of these models/indices in the future. Therefore, it would be good to settle on one working method because most likely some of them will vary in scale. I'd also like to use RMSE in a similar fashion to MAE if thats possible. Or is there a totally different or better way for doing the comparison? I'm definitely not that familiar with the world of statistics so any help is appreciated.

Thank you!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Far better, I would say, is to standardize/normalize the data, and then use the same metric of choice as is on all the models, as opposed to trying to adapt your metric of choice to the data.

The measures which you mentioned are all in the absolute scale, alternatively you could use a relative measure, such as MAPE, although it has its issues.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This might probably be the best option, but I'm not 100% that it's possible. This due to the first index/model using % of area covered by binary (0,1) pixels and therefore produces results on 0-100 scale (0-100% I guess). The second one takes the average pixel value (these pixels have the range of -1 to 1) within an area and therefore produces results on -1 to 1 scale. $\endgroup$
    – Jusba
    Jul 22, 2021 at 10:35
  • $\begingroup$ I'm thinking that It would maybe be possible to standardize those -1 to 1 pixels and then calculate the average pixel value, therefore producing normalized results. But would these then be comparable to the observed values or the values produced by the first index, which would be normalized? Could I then run ordinary MAE to compare these two? $\endgroup$
    – Jusba
    Jul 22, 2021 at 10:38
  • $\begingroup$ @Jusba You would first standardize/normalize both datasets such that all the variables are within [0,1] (in case of normalization), and then you can compare the models built on these transformed data sets using regular MAE. $\endgroup$
    – user2974951
    Jul 22, 2021 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.