What is required for neural network to approximate discontinuous function?

Question

I have coded a neural network with 1 hidden layer and 1 numerical output. No biases.

By appropiate choice of the activation function, I can easily approximate some continuous function.

However, even something as simple as

f(x) = 50 if x >= 100, and 25 otherwise

I cannot approximate using my neural net. I've tried changing learning rate, and also number of hidden notes, but it simply won't converge to it.

Why does it do so poorly at this incredibly simple function? What can I do to make it converge? It is not just this function, but an other similarly non-continuous function.

Answer 1

Wikipedia provides a synopsis of the universal approximation theorem.

In the mathematical theory of artificial neural networks, the universal approximation theorem states that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $\mathbb{R}^n$, under mild assumptions on the activation function.

This theorem is the core justification for attempting to model complex, nonlinear phenomena using neural networks. Even though it is very flexible, it doesn't cover everything -- in this case, you've defined a discontinuous function, and the universal approximation theorem only extends to continuous functions.

I am not aware of a theorem which allows a neural network to approximate arbitrary, discontinuous functions.

---

Perhaps if you treated either case of your target variable as a categorical outcome and used cross-entropy loss you would have success approximating the decision boundary between the two cases.

Answer 2

In general, learning a discontinuous function is hard as the error around the point where the value jumps are usually large.

However, in practice, there are ways to get around the problem by making the data less abrupt. For example, in your case, the function jumps when x = 100. We can change it to function:

g(x) = 50, x >= 100

g(x) = 25, x < 0

We left the value for the function undefined when 0 <= x < 100. The function g(x) can be also viewed as a displacement of f(x):

g(x) = f(x), x >= 100

g(x) = f(x+100), x < 0

Learning g(x) is much easier as there are no abrupt jumps. The neural network is free to choose any value for g(x) when x is between 0 and 100. It is easy to see, one sigmoid function will be enough to describe the sampled data from g(x).

After g(x) is learned, it is trivial to convert it back to f(x).

This approach works when there are a limited number of know discontinuous points.

Answer 3

Without knowing too much more about your specific implementation, what I can say is that neural networks by and large employ back-propagation/gradient descent in order to tune the network and improve the output's fit to the training data. This means that the loss function is differentiable, in most cases. The function you are trying to approximate has a large jump discontinuity, which would mean that most standard loss functions used in the optimization step of your neural network are going to fail at the discontinuity, and depending on the particular loss function and the details of the optimization algorithm used, will deal with this failure in any of various ways. If you were to apply a sigmoid layer at the end, for instance, you might be able to coax behavior that reasonably approximates the two lines "far enough" away from the discontinuity, but the output would still produce a continuous function connecting the two flat lines.

Other approximation techniques will run into similar problems. You might want to check out the Gibbs Phenomenon (https://en.wikipedia.org/wiki/Gibbs_phenomenon) from Fourier analysis and 'ringing artifacts' occurring in signal processing in general.