I am trying to use a neural network to learn the below function. In total, I have 25 features and 19 outputs. The above image shows the distribution of two features with respect to one of the outputs. There are close to 200k examples, and the neural network contains 7 layers with 256 neurons having leaky relu as activation. The last layer is a linear layer. The problem is that the output is too steep and the neural network has a large error on peaks. How can I modify the target variables or neural networks such that it makes a small mean absolute error on the peaks (basically flatten the below error plot)? Below is the error plot
Things which I have tried but have not worked-
- Used log to transform target variables.
- Increasing the size of the neural network with the hope that a large enough neural network will fit anything but the training loss is still large on peaks
- Used mean absolute, mean squared and percentage loss