0
$\begingroup$

I aim for replicating an numerical (non stochastic) algorithm by a neural network. Therefore I have basically an unlimited amount of data and I wish that the network have an almost perfect fit in terms of MAPE.That means I wish:

  1. MAPE should be around 0.5 %
  2. Maximum absolute procentual deviation should not exceed 5%

The problem is that the algorithm outputs quite small values (of the order of 10^-4) and therefore minimizing the L2 or L1 error directly on them (or a scaled version) does not translate to good MAPE performance. Choosing MAPE as training loss didn't work well (as expected). I then logarithmized the values which helped a lot but MAPE is still around 4% and is unevenly distributed (A subset of smaller values have bigger MAPE). I find that unsatisfying and have the following questions:

  1. Are neural nets the appropriate method here? I have the feeling that neural nets just are not able to approximate a deterministic function without noise with high precision.
  2. Is there anything else I can do to get a better MAPE Performance?
  3. I had the idea to increase frequency of data samples on which MAPE is high. It seems to me that there should be literature on it but I lack the right search term. It reminds a little of active learning but I could't find relevant literature. Could you provide something like that or tell my why the idea is bad?

I am not sure if it is relevant but my architecture consisted of modules of the type [Linear, Relu, Batch_norm] with skip connections. I experimented with various depths and widths but I couldn't decrease the MAPE below 4% (on the training set). The data consists of 23 explanatory variables and the size of my (generated) dataset is 3 million.

Improve this question
$\endgroup$
5
  • $\begingroup$ You’re simulating the data, right? Then you should be able to say if your features really do explain the outcome as well as you hope. They might not. In the extreme, consider how this would go if the outcome were totally independent of the features. $\endgroup$
    – Dave
    Apr 12, 2021 at 10:16
  • $\begingroup$ @Dave yes, exactly. The numerical Algorithm is defined on R^23 . I sampled uniformly from a practically relevant subset, passed the values through the algorithm and saved it all in a dataframe. Therefore the features really should explain y perfectly. Furthermore I am talking about the performance on the training set and even there I just can't push the limit no matter how many hidden layers i have (Maximum I tried was [2048,1024,512,256,128,64,32,16]) $\endgroup$
    – Roman27
    Apr 12, 2021 at 10:27
  • $\begingroup$ (1) Why do you expect that using the MAPE as training loss does not work well? I would expect that this strategy should be the exact way to go. (2) If there is no noise and unlimited data, then you should be able to (over-)fit your data to a perfect fit and obtain 0% MAPE, so why don't you aim for that? Actually, then it would not even make sense to use MAPE - using MSE would yield the same perfect fit and be better behaved numerically. ... $\endgroup$
    – Stephan Kolassa
    Apr 12, 2021 at 12:05
  • $\begingroup$ ... In any case, you might want to look at What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? That minimizing L1 or L2 loss does not translate into good MAPEs is unsurprising in the presence of noise (!), see Kolassa (2020, IJF). Which is again why I would suggest using (a smoothed version of) the MAPE as a training loss. It looks like I am not understanding something. Please clarify. $\endgroup$
    – Stephan Kolassa
    Apr 12, 2021 at 12:07
  • $\begingroup$ @StephanKolassa thanks for the response. I tried both but as I said I couldn't achieve a near perfect fit. I don't know why. $\endgroup$
    – Roman27
    Apr 15, 2021 at 22:18

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.