1
$\begingroup$

I've made a neural network designed to do regression. However, my dataset is unbalanced, and the data in the smaller section of the dataset have very different target values than the target values in the bulk of the dataset (by orders of magnitude).

My network gets good results for the bulk of the data, but bad results in the tail. I would like to try to improve this as much as I can before I resort to making more data.

An idea is to modify the loss - ie, not use MSE. I imagine, that since the majority of the data is quite similar, this area is "swamping" the loss function. If we have two points, x=1 and y=0.001, even if when they are predicted the distance from the points are the same it can have a very different meaning. (If they are both .1 out, this will affect y "more"). Is it, therefore, sensible to write a loss function where the relative difference contributes? So instead of the loss summing over absolute differences, it sums over % differences.

It seems that MAPE is exactly what I am looking for - the loss function will treat all the data equally (though I understand there are other issues with this method). Am I correct in this assessment, and if so, are there any variants of MAPE that would be even better suited?

$\endgroup$
3
$\begingroup$

I think it would be worthwhile for you to read carefully through my answer to What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? (Yes, I'll admit I also wrote the question, because I believe my input is helpful.) In particular, pay attention to the last bullet point in that answer, and to the explanation that follows.

The solution in my opinion lies in understanding what a point forecast is, and what a point forecast error measure (PFEM) attempts to do.

  • A point forecast is a one-number summary of a predictive distribution. This distribution is very often not explicit, but it always lurks in the background, and understanding is is fundamental. Of course, there are many different possible one-number summaries of a distribution: the mean, the median, the 90% (or any other) quantile, the mode, ... All these are so-called functionals of the predictive distribution.
  • What does the MAPE, the MSE or any other PFEM do? It evaluates how close our estimate for the target functional gets to the actual functional. The MSE is minimized in expectation if you get the correct expected value of the future distribution. The MAE is minimized in expectation by the correct median. And so forth.

Here is the problem: if you want an unbiased expectation forecast, then the MAPE will mislead you, because it is minimized in expectation by quite a different functional than the expectation. You will need to use the (R)MSE. If you think your forecasts get "better" if you use the MAPE, then I would suspect that changing the PFEM without understanding these relationships may be a case of walking down a blind alley.

You don't tell us what distributional assumptions your NN architecture makes. If your NN models changing expectations but constant variance (which I strongly suspect), then it simply does not correctly model the variation in your data distributions. (Higher means usually come with higher variances, especially if the differences in means are as large as you explain. If variance was constant, you would not have a problem in using the MSE.) If you change your error measure to the MAPE without addressing this point, then you are hearing weird noises from your car engine and thumping the hood until the weird noises cease. For the moment. It's better to understand where the noises come from.

I would suggest making sure you are using a method that can deal with changes in data distributions, and that outputs full densities. Then assess these densities using proper scoring rules.

$\endgroup$
5
  • $\begingroup$ thank you. This is a great answer, very informative, lots to think about! I have a few follow-up questions: when you say "expectation by quite a different functional than the expectation" are you referring to the points you made in your other answer about the asymmetry and high coefficient of variation problems? $\endgroup$
    – Andrew
    Jun 15 '20 at 17:41
  • $\begingroup$ Then, looking at your last paragraph, if I plot the output of the network (say distributions of %error and residuals) what am I looking for to justify attempting to use MAPE? If my output using MSE is not constant (residuals increase as target increases) is that evidence that MSE is the correct choice of loss? You said if the model is constant it "does not correctly model the variation in your data distributions", or am I misinterpreting? $\endgroup$
    – Andrew
    Jun 15 '20 at 17:43
  • $\begingroup$ Re your first comment: note that I wrote "it is minimized in expectation" and "by quite a different functional than the expectation". On the one hand, we have the expected value of the future distribution, which we may want to elicit (i.e., an unbiased expectation forecast), this is the second "expectation". The first "expectation" refers to the fact that the property of (say) the MAE to be minimized by the median of the future distribution is something that of course only holds in expectation. For any finite sample, we will find some other MAE minimizer than the true median. $\endgroup$ Jun 16 '20 at 14:00
  • $\begingroup$ Unfortunately, this is confusing. Sorry, but I don't see how to be precise, correct and non-confusing about this... $\endgroup$ Jun 16 '20 at 14:00
  • $\begingroup$ Re your second comment: even if your model outputs non-constant point forecasts, it may still not be modeling non-constant variances. I am arguing that it then still doesn't correctly model the future distributions. You may still want to use the MAPE - if the forecasts are more useful to you, by all means do so (we don't forecast for the fun of it, but because we use the forecasts for something) - I just believe it's good to keep the influence of the future distribution in the back of your head. $\endgroup$ Jun 16 '20 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.