0
$\begingroup$

If I have used MASE to calculate non time-series data error (as described by Dr. Rob Hyndman here), how can I know if the score received is good or not? Since it is not a time-series, a random walk naive model is irrelevant here, and the threshold of 1 which is usually used doesn't seems to be a good comparison. Is it correct? if yes - what should be the consideration to decide what score is good or not? if not - why?

BTW it seems that even for time series data, a score of 1 and bellow is not always an achievable target as describe here - so how can you know what is considered to be sufficient?

$\endgroup$
0
+100
$\begingroup$

Based on his textbook for a non-seasonal time series, a useful way to define a scaled error uses naïve forecasts: $$ q_{j} = \frac{\displaystyle e_{j}}{\displaystyle\frac{1}{T-1}\sum_{i=1}^T |y_t-y_{t-1}|} $$

Acoording to his answer here, it is possible to scale the data using the mean as the base forecast. If $e_j$ denotes a prediction error on the test data, then the scaled errors $q_j$ are: $$ q_{j} = \frac{\displaystyle e_{j}}{\displaystyle\frac{1}{N}\sum_{i=1}^N |Y_i-\bar{Y}|} $$ where $y_1,\dots,y_N$ denotes the training data.

Note here that $Y_{t-1}$ is replaced with $\bar{Y}$

The MASE compares "your" forecast against a naive benchmark forecast calculated in-sample due to the MASE denominator is calculated in-sample, not in the holdout sample. The MASE is a metric for comparing errors to a user-defined baseline:

$$ MASE= mean(|q_{j}|) $$

For interpretation of MSE which is not an easy job please have a look at this earlier thread on interpreting the MASE might be helpful from @Stephan Kolassa

My recommendation:

  • Check this answer to not confused about the calculation of MSAE and MAE
  • if you have a single series, then the MASE is less informative than the MAE, since the MASE is simply the MAE scaled by a factor that does not depend on the forecast. The MASE makes sense once you have multiple series on different levels, where you can't very well compare "raw" MAEs.

Note: the scaling statistic $Q$:

$$ \text{MASE}=\text{MAE}/Q $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.