How to learn steep functions using neural network?

Question

I am trying to use a neural network to learn the below function. In total, I have 25 features and 19 outputs. The above image shows the distribution of two features with respect to one of the outputs. There are close to 200k examples, and the neural network contains 7 layers with 256 neurons having leaky relu as activation. The last layer is a linear layer. The problem is that the output is too steep and the neural network has a large error on peaks. How can I modify the target variables or neural networks such that it makes a small mean absolute error on the peaks (basically flatten the below error plot)? Below is the error plot

Things which I have tried but have not worked-

1. Used log to transform target variables.
2. Increasing the size of the neural network with the hope that a large enough neural network will fit anything but the training loss is still large on peaks
3. Used mean absolute, mean squared and percentage loss

Answer 1

Some years ago, I spent a lot of time solving a similar problem.

Let me say that accurate function approximation with neural nets can be really hard.

First of all, in my experience, standard optimization methods used in deep learning (e.g., for image recognition and language modelling) are not very well-suited to this kind of problem, especially if the capacity of the network (number of neurons and layers) is not very large.

I would start by trying to reproduce the identity mapping with the algorithm you are currently using: take your 25 input features and use them as both input and output and see what happens. You might easily find that your training algorithm is not even able to accurately reproduce the identity mapping.

Another problem might be caused by multicollinearity in the outputs, which can seriously impair the convergence properties of the training algorithm (basically, you have ill-conditioned matrices somewhere in the net). Try to train a separate network for each of the 19 outputs and see what happens.

Try custom parameter updating rules. The one that worked best for me was parameter update = -sign(gradient) * indicator(gradient and momentum have the same sign). I found this in a paper that tried thousands of rules. This rule was the best one. Updating only one layer per iteration may also help.

Increase the batch size as the loss decreases. With your problem, I would say, start from 50 and increase to something around 10000 in the end.

Decrease the learning rate as the loss decreases.