Skip to main content
Link for error metrics
Source Link
NG_21
  • 1.6k
  • 4
  • 17
  • 25

It's quite common to have conflicting values for different error metrics like MAE, ME, RMSE, MAPE. There are two main reasons behind that

  1. The way the formulas for those metrics are defined.
  2. The nature of your data itself.

While you can't do much with regard to first reason.You should really examine the nature of your data, to decide on selecting the error metrics to compare different models.

Lets take a very basic example, suppose your data series has few values which are zero or very small ( close to zero), In that case MAPE( Mean Absolute Percentage Error) would not be a correct metric to compare your mode.Even though your forecast might be reasonably good, MAPE(both in sample /out of sample) could be very high due to those zero values.

Bottom Line : Examine your data.

The Problems I deal with : The previous approach works quite good if we have handful of time series. But what if we are dealing with selecting forecasting models for 300-500 time series ( a common case in supply chain, retail, manufacturing). Well the best approach is to calculate all the metrics for every model like ME, RMSE, MAPE, MPE, MAE, MASE( Mean Absolute Scaled Error) or any other metrics which is used in your domain.

Select the model which gives low values for most of the error metrics. This process can be automated.

For a list of modern error metrics you can refer [here](http://robjhyndman.com/papers/mase.pdf "

It's quite common to have conflicting values for different error metrics like MAE, ME, RMSE, MAPE. There are two main reasons behind that

  1. The way the formulas for those metrics are defined.
  2. The nature of your data itself.

While you can't do much with regard to first reason.You should really examine the nature of your data, to decide on selecting the error metrics to compare different models.

Lets take a very basic example, suppose your data series has few values which are zero or very small ( close to zero), In that case MAPE( Mean Absolute Percentage Error) would not be a correct metric to compare your mode.Even though your forecast might be reasonably good, MAPE(both in sample /out of sample) could be very high due to those zero values.

Bottom Line : Examine your data.

The Problems I deal with : The previous approach works quite good if we have handful of time series. But what if we are dealing with selecting forecasting models for 300-500 time series ( a common case in supply chain, retail, manufacturing). Well the best approach is to calculate all the metrics for every model like ME, RMSE, MAPE, MPE, MAE, MASE( Mean Absolute Scaled Error) or any other metrics which is used in your domain.

Select the model which gives low values for most of the error metrics. This process can be automated.

It's quite common to have conflicting values for different error metrics like MAE, ME, RMSE, MAPE. There are two main reasons behind that

  1. The way the formulas for those metrics are defined.
  2. The nature of your data itself.

While you can't do much with regard to first reason.You should really examine the nature of your data, to decide on selecting the error metrics to compare different models.

Lets take a very basic example, suppose your data series has few values which are zero or very small ( close to zero), In that case MAPE( Mean Absolute Percentage Error) would not be a correct metric to compare your mode.Even though your forecast might be reasonably good, MAPE(both in sample /out of sample) could be very high due to those zero values.

Bottom Line : Examine your data.

The Problems I deal with : The previous approach works quite good if we have handful of time series. But what if we are dealing with selecting forecasting models for 300-500 time series ( a common case in supply chain, retail, manufacturing). Well the best approach is to calculate all the metrics for every model like ME, RMSE, MAPE, MPE, MAE, MASE( Mean Absolute Scaled Error) or any other metrics which is used in your domain.

Select the model which gives low values for most of the error metrics. This process can be automated.

For a list of modern error metrics you can refer [here](http://robjhyndman.com/papers/mase.pdf "

Source Link
NG_21
  • 1.6k
  • 4
  • 17
  • 25

It's quite common to have conflicting values for different error metrics like MAE, ME, RMSE, MAPE. There are two main reasons behind that

  1. The way the formulas for those metrics are defined.
  2. The nature of your data itself.

While you can't do much with regard to first reason.You should really examine the nature of your data, to decide on selecting the error metrics to compare different models.

Lets take a very basic example, suppose your data series has few values which are zero or very small ( close to zero), In that case MAPE( Mean Absolute Percentage Error) would not be a correct metric to compare your mode.Even though your forecast might be reasonably good, MAPE(both in sample /out of sample) could be very high due to those zero values.

Bottom Line : Examine your data.

The Problems I deal with : The previous approach works quite good if we have handful of time series. But what if we are dealing with selecting forecasting models for 300-500 time series ( a common case in supply chain, retail, manufacturing). Well the best approach is to calculate all the metrics for every model like ME, RMSE, MAPE, MPE, MAE, MASE( Mean Absolute Scaled Error) or any other metrics which is used in your domain.

Select the model which gives low values for most of the error metrics. This process can be automated.