I am using the normalized mean absolute error metric for evaluating my results. The data I use is in time-series form. Their trend may be increasing or decreasing over time. All the values are positive at first and in different scales and ranges. I standardized my data in this form: first and for all samples, subtracting each time point from its baseline value and then divided them by the std of the baseline values(of all samples). Then, using the NMAE formula for my prediction. But now, I am getting some negative values for the normalized mean absolute error metric. I don't know what does it mean?
7
-
$\begingroup$ "subtracted each time point from their baseline values and then divided them by the std. of the baseline values" - at what point did you change these values (which will be positive and negative) to absolute values? $\endgroup$– zbicyclistNov 5, 2020 at 4:32
-
$\begingroup$ I have used this formula as the metric: \begin{equation} NMAE = \frac{\sum_{i=1}^{N} \mid Y_{pred}-Y_{real} \mid}{\sum_{i=1}^{N} Y_{real} } \end{equation} $\endgroup$– elldoraNov 14, 2020 at 8:03
-
$\begingroup$ The negative value for this metric depends on the denominator of the NMAE formula. I think after the standardization (described above), there could be appeared some negative values in the data and their summation as well. Now, I don't know the interpretation of the negative value for this metric? From the MAE we know as much as it is closer to the zero, the model performance is more acceptable. Should I consider this metric's absolute value as well and interpret it similarly? $\endgroup$– elldoraNov 14, 2020 at 8:33
-
$\begingroup$ If your Y_real values aren't all positive, this measure isn't appropriate. Use MAE, although that still gives you the problem of comparing across series. $\endgroup$– zbicyclistNov 15, 2020 at 15:13
-
$\begingroup$ One possible way to deal with this would be to standardize each series around a mean of 500 and a standard deviation of 100, so all the values would be positive. [picking 500 and 100 is arbitrary -- that's just the idea behind SAT college scores] $\endgroup$– zbicyclistNov 15, 2020 at 15:14
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
You could use the absolute values of Yreal, since the interest here is the magnitude of MAE with respect to the magnitude of actual values (independently of the sign).
$\begingroup$
$\endgroup$
The comments give the normalized mean absolute (NMAE) error as follows.
$$ NMAE = \dfrac{ \overset{N}{\underset{i = 1}{\sum}}\left\vert Y_{pred} - Y_{real} \right\vert}{ \overset{N}{\underset{i = 1}{\sum}}Y_{real} } $$
Since the numerator is a sum of absolute values, the numerator cannot be less than zero. Consequently, if the overall $NMAE$ is less than zero, then the denominator must be less than zero, meaning that these $Y_{real}$ values must have an arithmetic mean (the usual mean) less than zero. That does not tell me much, but if you know the way you have constructed the $Y_{real}$ precludes a mean below zero, then this would be an indication of a bug in your implementation code that needs to be fixed.
Other than that, I do not see much that can be concluded, though I have my doubts that the given formula represents a useful calculation. Sure, we can write a software function to perform the calculation, stick data into the function, and get a result, but I do not see a reason to care about such a calculation. The searching I have done for normalized mean absolute error seems to put an absolute value in the denominator.
$$ NMAE = \dfrac{ \overset{N}{\underset{i = 1}{\sum}}\left\vert Y_{pred} - Y_{real} \right\vert}{ \overset{N}{\underset{i = 1}{\sum}}\left\vert Y_{real}\right\vert } $$
I am not even so sold on this being especially useful, unless you can put it in the context I discuss here where the $Y_{real}$ have a mean (maybe median for absolute errors instead of squared) of zero, but this formula definitely cannot yield values less than zero.
