1
$\begingroup$

I have following dataframe:

  Impute Technique   Mean    Std   SMAPE
1   EINSTEIGER_ROH   0.00   0.00     NaN
6             MEAN  20.18   8.83  123.40
5      KNN_IMPUTER  20.30   8.79  126.56
7           MEDIAN  22.12   9.30  112.35
3      INTERPOLATE  28.91  10.56  120.24
2            FFILL  35.67  11.27  112.86
0            BFILL  41.01  22.54  112.98

I tried different Imputing techniques (for missing values) and wanted to evaluate which one worked best on average. In column "Impute Technique" you can see the different methods, and column "Mean" represents the Mean Average Error, "Std" the mean standard deviation and "SMAPE" the results from a SMAPE calculation (I couldn't use MAPE because of 0 values in y_pred and y_true).

Now I can't really explain why e.g. for the "Mean" column the "MEAN" method worked best but for the "SMAPE" column the "MEDIAN" method worked best. How can this be explained?

For calculating the SMAPE, I used following function:

def smape(A, F):
    return 100/len(A) * np.sum(2 * np.abs(F - A) / (np.abs(A) + np.abs(F)))

Thanks and regards!

$\endgroup$
5
  • 1
    $\begingroup$ I believe my answer in the proposed duplicate answers your question: the sMAPE and the MAE (in your Mean column) elicit different functionals of the unknown distribution, i.e., they reward different ways of summarizing (implicit) knowledge about this distribution. Thus, for any given imputed value, or imputation algorithm, they will differ and essentially tell you how close these are getting to the functional this particular evaluation measure prefers. I have a number of threads here on this, most cast in forecasting rather than imputation terminology, but the issues are identical. $\endgroup$ Commented Nov 13, 2023 at 16:01
  • $\begingroup$ Here is a thread explicitly on the sMAPE: Minimizing symmetric mean absolute percentage error (SMAPE) Essentially, if you want to minimize the MAE, you should be using a different imputation technique than if you want to minimize the sMAPE. See Kolassa (2020). $\endgroup$ Commented Nov 13, 2023 at 16:02
  • $\begingroup$ thanks for answering! I will have a look. Originally I wanted to calculate MAPE but as there were many Zeros and I can't divide by 0, I switched to SMAPE. Do you think this makes sense? $\endgroup$ Commented Nov 13, 2023 at 16:07
  • $\begingroup$ My recommendation is always to first figure out which functional you want to elicit. Do you want to impute the conditional expectation? Use the MSE. Do you want the conditional median? Use the MAE. Any variant on the MAPE (i.e., also the sMAPE and the wMAPE) will tend to reward optimal imputations that are lower than the conditional expectation. If that is what you want, great. Here and here are examples how strong the effect can be. $\endgroup$ Commented Nov 13, 2023 at 16:16
  • $\begingroup$ This may also be helpful. Fun fact about the sMAPE: it at least does not divide by zero, but whenever you have an actual zero, your imputation will be penalized by 200%, regardless of the value of your imputation. I don't think this is so much better than the MAPE, where you at least find out that something is problematic with zeros. $\endgroup$ Commented Nov 13, 2023 at 16:40

0

Browse other questions tagged or ask your own question.