actually my study is about to choose best imputation techniques. Im comparing all the techniques using Root Mean Square Error (RMSE), Mean Aabsolute Error (MAE) and correlation (R). My problem is example imputation at Station . Method A has lower RMSE compare to method B and the R value is 0.7. Method B has lower MAE value and R value is also 0.7 Is it both method is consider the best?

  • $\begingroup$ What are you comparing? The literature on imputation suggests evaluating the bias between the pre-imputation means and std devs on the nonmissing data versus the post-imputation means and std devs on the nonmissing plus imputed data. $\endgroup$ Apr 3 '17 at 11:29

Consider the difference between MAE and RMSE. While both aim to measure 'fit' in some sense, they have slightly different measures of what 'fit' actually is. RMSE penalises estimates further from the true value more than if a model produced several smaller errors, even though the total MAE may be the same.

For example see the below (python), where moving from y_pred_1 to y_pred_2 improves RMSE but MAE is worse.

from sklearn import metrics
from math import sqrt

y_pred_1 = [6,4,3,2]
y_pred_2 = [3,5,4,4]
y_actual = [2,4,3,2]

metrics.mean_absolute_error(y_actual, y_pred_1) # 1.0
metrics.mean_absolute_error(y_actual, y_pred_2) # 1.25

sqrt(metrics.mean_squared_error(y_actual, y_pred_1)) # 2.0
sqrt(metrics.mean_squared_error(y_actual, y_pred_2)) # 1.32

Which of the measures is more useful depends on the specific task you're modelling. If it's unclear which metric is more useful, then I'd have said it's valid to report both, and try to understand why each of your imputation methods affects things the way they do.

Looking briefly at rainfall prediction literature, R^2 seems to be used (which is just your R value squared) at times e.g. Statistical models for predicting rainfall in the Caribbean. Although I also saw RMSE and MAPE used e.g. Evaluation of the Liu model for predicting rainfall. So no easy answer!

  • $\begingroup$ Im trying to impute the rainfall data. $\endgroup$
    – Rosbert
    Apr 3 '17 at 10:36
  • $\begingroup$ see updated answer $\endgroup$
    – tea_pea
    Apr 3 '17 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.