Is Median Absolute Percentage Error useless?

Question

I'm working on a project focused on pricing houses. Looking online I see a lot of works and companies providing the performances of their model using the median instead of the mean (see for example Zillow). And to me this makes total sense, since its a statistic less influenced by outliers (in my scenario big and particular houses).

Also since I would like to have a greater interpretability of the performance of my model I used the Absolute Percentage Error, instead of the squared error.

In the end what I ended up using is the Median Absolute Percentage Error, and with my surprise I discovered that is not a common metrics, it doesn't have a wikipedia page, few sites provide an explanation of what it is, sklearn doesn't even have it as a metrics inside the sklearn.metrics._regression.py module.

And my question is: why?

Fixing it was a no brainer, really 2 line of code and I created my own version of it and I can easily run my model with this new metric, but I was wondering if maybe I'm missing something, like if its completely useless or maybe there are other metrics that works better.

If anyone could give me some insights that would be great!

Answer 1

I would be very careful about percentage errors, especially in the context of skewed distributions in the outcome (more precisely: skewed error distributions).

Look at each separate prediction. You evaluate it using an Absolute Percentage Error. This metric will prefer underpredictions, and especially so in the case of skewed error distribtutions. See What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?, and note that the argument really applies to the raw APE, whether we summarize the APEs using the mean or the median.

Summarizing these separate APEs using the median instead of the mean (or using trimming, per Christian's answer) will attenuate the problem, but it won't solve it. "Optimal" predictions will still be biased low. You can simulate this by running an analysis like in the post linked to above: simulate skewed data whose distribution you know (this would stand in for the unknown error distribution in your application), and see which one-number summary will minimize the median APE.

If minimizing the APE is what you really want, and bias is not a problem for you, then by all means, go ahead. I just can't think of a business problem that would be better addressed by an APE-optimal forecast rather than an unbiased expectation forecast. As such, I would say that the "better interpretability" of APEs is a mirage.

Answer 2

Be careful with the median for performance metrics! Robustness to a small number of outliers is a good thing in most cases, but if you use the median a method may look good that in fact gives you a bad result in, say, 30 or 40% of the cases, and that's mostly not appropriate. I have used one-sided upper 10% trimmed means in such cases (probably not implemented in standard packages either) to express that I'm happy if I can have a good fit in 90% of the cases even if up to 10% predictions (or whatever performance you are measuring) are bad, but I don't want to tolerate more than that. It depends on the specific situation though.

Other than that, in principle your idea makes some sense, and the fact that it isn't implemented anywhere (or at least not where you looked) could be explained by the fact that many people don't think very much about the specific performance metric and are happy with what is available by default, which is often motivated by certain mathematical considerations that may be rather irrelevant for the application in hand. Many fairly simple but nonstandard things that can be useful in a certain situation are not implemented in standard packages. It's always good to question ones own ideas, but the bare fact that it isn't implemented doesn't mean there's something fundamentally wrong with it.

By the way here's something we've written some time ago: Some thoughts about the design of loss functions

Answer 3

I've spent a few years building real estate price regressors, which are known as "AVMs" (Automated Valuation Models). A few comments:

• Yes, "median absolute percentage error" is both a reasonable metric and one that gets used in practice. (And in an instance of catastrophic acronym failure, "MAPE" can refer either to "Mean" or "Median" absolute percent error.)
• Measuring "median absolute percentage error" is very strongly related to measuring "median absolute log error", and the latter has some advantages.
  • If you're considering log error, then deviations become symmetric. (This addresses quarague's observation about overestimating or underestimating by 2x.)
  • Implementing this metric should be very easy-- you just take the log() of everything and evaluate the MAE().
  • The logarithm also suppresses very large outliers (e.g., absurdly expensive mansions), which is one of the problems with examining real estate data. This can have both modeling and numerical stability advantages. (There are also non-arms-length real estate deals, which lead to outliers that are very small, but these are frequently easier to detect.)
• There are companies that provide AVMs for banks and so forth; there are also companies that exist solely to evaluate the first set of companies. If you're trying to understand standard evaluation methodology in this space, you may want to examine the latter. To get started, maybe check out AVMetrics. (I have no connection with that company, by the way, aside from having read one of their analyses.)

Answer 4

The potential issue with absolute percentage error is that it is not symmetric with respect to over and underestimating. If you overestimate by a factor of 2 you will get an error of 100%, if you underestimate by a factor of 2 you will get an error of only 50%.

If you model is so good that most estimates are very close to the true value this effect becomes relatively small but if some estimates are way off this will impact your model choices. A model that occasionally make severe underestimates but no overestimates will look better than a model that occasionally makes severe overestimates.

Whether this happens in your case and if it does whether it is a problem depends on your specific situation but it is something to be aware off when looking at absolute percentage errors.

Answer 5

TLDR; Here is a point of view in terms of what the median/mean tell about the behaviour of the tails. The median gives little information while the mean does.

---

A related question is Chebychev-like inequality based on the median absolute deviation (about the median)

I answered that question while understanding the problem to be about the mean absolute deviation. The reason that I did that is because the mean makes much more sense in relation to Chebychev-like inequalities. A problem with the median is that it only relates to a single point on the distribution curve

$$\text{median}(X) = x:F(x) = 0.5$$

The median tells little about the entire distribution and what the tail of the distribution does. The median can be even zero if more than 50% of the deviation is zero.

The mean on the other hand, gives a more weighted information about the entire distribution and includes the tails.

$$\text{mean}(X) = 1- \int_0^\infty F(x) dx$$

Let's look at a few curves with median or mean equal to 1.

The red curves have an average of 1 and will be restricted to be below the black curve 1/x.

The blue curves have a median of 1 and will only pass the point (1,0.5), but further from that they can be any shape.