3
$\begingroup$

I have a time series data that is not stationary (with trend and seasonal components) so in order to make it stationary, I've applied a difference transform of 1. Due to this effect, some negative values appeared. I am applying cross validation and I want to calculate error metrics such as MAPE/sMAPE on my validation sets, but due to the existence of negative values, MAPE & sMAPE are getting greater than 100%. I know that MAPE has many pitfalls but I need a percentage error metric.

A similar issue is asked here in this thread, but I am still not sure about this.

  • Question 1: How can I avoid values greater than 100% when negative values exist in my data ?
  • Question 2: Are there alternatives to MAPE/sMAPE that combat this issue ? perhaps suited also for time series problems

I am using the following code snippets:

def mean_absolute_percentage_error(y_true, y_pred):
    y_true, y_pred = np.array(y_true), np.array(y_pred)
    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100

def symmetric_mean_absolute_percentage_error(y_true, y_pred):
    return 100 / len(y_true) * np.sum(2 * np.abs(y_pred - y_true) / (np.abs(y_true) + np.abs(y_pred)))
Improve this question
$\endgroup$
5
  • $\begingroup$ It seems that the negative values come about because you are applying your metric to the differenced series; what happens if you apply it to the undifferenced series, which is the one your customers (or you) are more likely to be interested in? $\endgroup$
    – jbowman
    Aug 27, 2019 at 18:42
  • $\begingroup$ @jbowman Thank you for your reply. Thats true. We are actually interested in applying it on the differenced data in order to get rid of non-stationarity. This of course does not happen when the data is not differenced. $\endgroup$
    – Perl Del Rey
    Aug 27, 2019 at 18:48
  • $\begingroup$ I understand that you DO want to difference the data, but why apply a percentage-based metric to the differences instead of the original series? If you have to, though, I'd take the denominator to be the absolute value, so it becomes in effect "change in $y$" rather than "signed change in $y$". (Naturally the numerator needs to take the sign into account.) $\endgroup$
    – jbowman
    Aug 27, 2019 at 18:52
  • $\begingroup$ @jbowman Thank you!! Isn't it a good practice to apply all metrics to the differenced data rather than the non-stationary one ? $\endgroup$
    – Perl Del Rey
    Aug 29, 2019 at 7:38
  • $\begingroup$ Well it depends. If you want to know how well your model performs, differencing is part of that, and you want to calculate metrics for the undifferenced series. For example, if I'm trying to model weight loss, and I build a model of weight loss based on the last 50+ days of daily data, there's going to be a difference term there, but what I care about is, e.g., the percentage error of the model when predicting future weight, not the percentage error of change in weight, since health is related to absolute weight. (Maybe this isn't a good example, though.) $\endgroup$
    – jbowman
    Aug 29, 2019 at 14:51

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.