0
$\begingroup$

Will it be correct to use Mean absolute scaled error in non time series data? I've got a set which contains a lot of zeros, so errors like MAPE can not be used here. MASE based on difference between yt and yt-1 (formula like here), but I don't have time series, just observations of some activity of 200 people (independent), so maybe I can just use difference based on second and first person and so on? If not, do you know any percentage measures that I can use in that kind of set?

$\endgroup$
2
$\begingroup$

(The standard version of) MASE only works for time series. The purpose of the $Y_t$ - $Y_{t-1}$ term in the denominator is that it represents a naive forecast, that it is your best guess for $Y_t$ is $Y_{t-1}$.

In your case, your data is non-sequential and more importantly it is independent, so the idea of using $I_{n-1}$ as an estimator for $I_n$ doesn't really make sense (why $I_{n-1}$? Why not $I_{n-2}$ or $I_{n-199}$?)

To answer your 2nd question, it depends on what error you are trying to measures exactly ?

For dealing with the issue of zeros, a possible alternative for MAPE is sMAPE (Symmetric Mean Absolute Percentage Error). That might be a good option for you.


Addressing the comment:

  • When $Y_{emp}$ and $Y_{pred}$ are both equal to zero, than you set the sMAPE to 0 (since they are equal to each other, there is no error) (add an if clause to your calculation). Many software packages do that already, even with MAPE.
  • For the second problem, you can use the modified version of the sMAPE:

$$\text{sMAPE} = \frac{1}{N}\sum \frac{|y_{emp} -y_{pred}|}{{|y_{emp}|+|y_{pred}|}}$$

This gives you a 100% error in cases like you mentioned ($Y_{emp} = 0$ and $Y_{pred}=0.005$)

So the modified version of sMAPE could be:

$$\text{sMAPE} = \begin{cases} 0, & \text{ if } Y_{emp} = 0 & and & Y_{pred}=0 \\ \frac{1}{N}\sum \frac{|y_{emp} -y_{pred}|}{{|y_{emp}|+|y_{pred}|}} & \text{otherwise} \\ \end{cases} $$

If even 100% is too high a threshold, you can apply some sort of log transformation.


Since I first wrote this answer, I noticed elsewhere in this forum that Rob Hyndman (the original author of MASE) proposes a modified version of MASE for non-time series data. See here

If $e_j$ denotes a prediction error on the test data, then the scaled errors are $$ q_{j} = \frac{\displaystyle e_{j}}{\displaystyle\frac{1}{N}\sum_{i=1}^N |Y_i-\bar{Y}|}. $$ where $y_1,\dots,y_N$ denotes the training data.

Note here that $Y_{t-1}$ is replaced with $\bar{Y}$.

In the time series case, this would mean that the prediction is being compared to the predictions produced by a mean forecast instead of a naive forecast (as it is in the standard formulation of MASE).

$\endgroup$
  • $\begingroup$ I 've read about this measure but i can't use this because of situation when I have $Y_{emp}$ =0 and $Y_{pred}$ =0, then this measure is bad. Also When I check only freq, when $P_{emp}$ =0 and $P_{pred}$ =0,0005 then this measure going to its top (200%). I need error which can by comparable and don't have problems like I texted above. $\endgroup$ – Roberto Jan 9 '18 at 17:14
  • $\begingroup$ @Roberto see my edits. $\endgroup$ – Skander H. Jan 9 '18 at 17:45
  • $\begingroup$ Well that makes sens for probabilities. Thank You for your help. Also I am looking on the last formula in the wikipedia you mentioned and I think it solve a problem to both. Am I right? $\endgroup$ – Roberto Jan 9 '18 at 18:00
  • $\begingroup$ @Roberto the last formula in the wikipedia page - log(F/A) - requires strictly positive data and you said you have a lot of zeros? Also it would lead to cases where the error is negative for F < A . $\endgroup$ – Skander H. Jan 9 '18 at 18:13
  • $\begingroup$ I meant SUM(abs[F-A]) / SUM(A+F). And yes, 80% of y in my set equal zero. $\endgroup$ – Roberto Jan 9 '18 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.