Is this the correct place to use MAPE as a loss function?

Question

I've made a neural network designed to do regression. However, my dataset is unbalanced, and the data in the smaller section of the dataset have very different target values than the target values in the bulk of the dataset (by orders of magnitude).

My network gets good results for the bulk of the data, but bad results in the tail. I would like to try to improve this as much as I can before I resort to making more data.

An idea is to modify the loss - ie, not use MSE. I imagine, that since the majority of the data is quite similar, this area is "swamping" the loss function. If we have two points, x=1 and y=0.001, even if when they are predicted the distance from the points are the same it can have a very different meaning. (If they are both .1 out, this will affect y "more"). Is it, therefore, sensible to write a loss function where the relative difference contributes? So instead of the loss summing over absolute differences, it sums over % differences.

It seems that MAPE is exactly what I am looking for - the loss function will treat all the data equally (though I understand there are other issues with this method). Am I correct in this assessment, and if so, are there any variants of MAPE that would be even better suited?

Answer 1

I think it would be worthwhile for you to read carefully through my answer to What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? (Yes, I'll admit I also wrote the question, because I believe my input is helpful.) In particular, pay attention to the last bullet point in that answer, and to the explanation that follows.

The solution in my opinion lies in understanding what a point forecast is, and what a point forecast error measure (PFEM) attempts to do.

• A point forecast is a one-number summary of a predictive distribution. This distribution is very often not explicit, but it always lurks in the background, and understanding is is fundamental. Of course, there are many different possible one-number summaries of a distribution: the mean, the median, the 90% (or any other) quantile, the mode, ... All these are so-called functionals of the predictive distribution.
• What does the MAPE, the MSE or any other PFEM do? It evaluates how close our estimate for the target functional gets to the actual functional. The MSE is minimized in expectation if you get the correct expected value of the future distribution. The MAE is minimized in expectation by the correct median. And so forth.

Here is the problem: if you want an unbiased expectation forecast, then the MAPE will mislead you, because it is minimized in expectation by quite a different functional than the expectation. You will need to use the (R)MSE. If you think your forecasts get "better" if you use the MAPE, then I would suspect that changing the PFEM without understanding these relationships may be a case of walking down a blind alley.

You don't tell us what distributional assumptions your NN architecture makes. If your NN models changing expectations but constant variance (which I strongly suspect), then it simply does not correctly model the variation in your data distributions. (Higher means usually come with higher variances, especially if the differences in means are as large as you explain. If variance was constant, you would not have a problem in using the MSE.) If you change your error measure to the MAPE without addressing this point, then you are hearing weird noises from your car engine and thumping the hood until the weird noises cease. For the moment. It's better to understand where the noises come from.

I would suggest making sure you are using a method that can deal with changes in data distributions, and that outputs full densities. Then assess these densities using proper scoring rules.