What is required for neural network to approximate discontinuous function?

Question

I have coded a neural network with 1 hidden layer and 1 numerical output. No biases.

By appropiate choice of the activation function, I can easily approximate some continuous function.

However, even something as simple as

f(x) = 50 if x >= 100, and 25 otherwise

I cannot approximate using my neural net. I've tried changing learning rate, and also number of hidden notes, but it simply won't converge to it.

Why does it do so poorly at this incredibly simple function? What can I do to make it converge? It is not just this function, but an other similarly non-continuous function.

Answer 1

Wikipedia provides a synopsis of the universal approximation theorem:

In the mathematical theory of artificial neural networks, the universal approximation theorem states that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $\mathbb{R}^n$, under mild assumptions on the activation function.

This theorem is the core justification for attempting to model complex, nonlinear phenomena using neural networks. Even though it is very flexible, it doesn't cover everything -- in this case, you've defined a discontinuous function, and the universal approximation theorem only extends to continuous functions.

I am not aware of a theorem which allows a neural network to approximate arbitrary, discontinuous functions.

Despite the impressive name, the universal approximation theorems (there are many) are less practical than one might like. Showing that there exists some approximating function with the desired qualities is very different from using some observed data to determine the weights and biases of that approximation.

---

But the universal approximation theorem is actually way more general than is needed for the specific function in OP's question:

$$ f(x) = \begin{cases} 50 & x \ge 100 \\ 25 & \text{otherwise}. \end{cases}$$ This function is just two horizontal lines with 1 discontinuity. As long as our approximation function is constrained to only predict values between 25 and 50 (the smallest and largest values of the function), we know that the largest absolute error will at most 25 (the largest possible difference between approximation and the true value). Bounding the maximum error is crucial because this bound is the precise sense in which a neural network approximates a function.

In practical terms, we need to identify a family of functions that can be parameterized such that specific parameter choices reduces the maximum error to a desired level.

For this specific problem, we just need a function that "looks like" two horizontal lines with some kind of "ramp" in the middle. We know that we want a "ramp" because we need to make predictions that are between 25 and 50, and because our approximation is continuous.

There are many such functions, but a common choice is a (scaled and shifted) sigmoid function. Just by reading off the statement of the problem, we can fill in nearly all of the free parameters:

$$\begin{align} \hat f(x) &= a + b \sigma(w(x-c)) \\ &= 25 + 25 \sigma(w(x-100)) \end{align}$$

This one-layer, one-neuron neural network seems like a decent approximation to the desired function. For instance, for $w=3$, we have

• $\hat f(99)= 26.18565\dots$
• $\hat f(101)= 48.81435\dots$

However, minimizing the loss by backpropagation will never reach a minimum because we can always improve this approximation by making $w$ larger. In the limit of $w$ tending towards infinity, the sigmoid approximation becomes the desired step function.

---

Suppose that instead of $$ f(x) = \begin{cases} 50 & x \ge 100 \\ 25 & \text{otherwise}, \end{cases}$$ we have the function $$ h(x) = \begin{cases} 1 & x \ge 100 \\ 0 & \text{otherwise}. \end{cases}$$ All we've done is shifted & rescale the function values. Restating the function in this way might be illuminating because $h$ is recognizable immediately as classification (using the conventional binary coding for the outputs). Moreover, we can see from its definition that this classification problem exhibits perfect separation, because there is some fixed constant above which we have only 1s and below which we have only 0s. So we know that the weight(s) $w$ will "drift away" towards infinity.

Answer 2

In general, learning a discontinuous function is hard as the error around the point where the value jumps are usually large.

However, in practice, there are ways to get around the problem by making the data less abrupt. For example, in your case, the function jumps when x = 100. We can change it to function:

g(x) = 50, x >= 100

g(x) = 25, x < 0

We left the value for the function undefined when 0 <= x < 100. The function g(x) can be also viewed as a displacement of f(x):

g(x) = f(x), x >= 100

g(x) = f(x+100), x < 0

Learning g(x) is much easier as there are no abrupt jumps. The neural network is free to choose any value for g(x) when x is between 0 and 100. It is easy to see, one sigmoid function will be enough to describe the sampled data from g(x).

After g(x) is learned, it is trivial to convert it back to f(x).

This approach works when there are a limited number of know discontinuous points.

Answer 3

Without knowing too much more about your specific implementation, what I can say is that neural networks by and large employ back-propagation/gradient descent in order to tune the network and improve the output's fit to the training data. This means that the loss function is differentiable, in most cases. The function you are trying to approximate has a large jump discontinuity, which would mean that most standard loss functions used in the optimization step of your neural network are going to fail at the discontinuity, and depending on the particular loss function and the details of the optimization algorithm used, will deal with this failure in any of various ways. If you were to apply a sigmoid layer at the end, for instance, you might be able to coax behavior that reasonably approximates the two lines "far enough" away from the discontinuity, but the output would still produce a continuous function connecting the two flat lines.

Other approximation techniques will run into similar problems. You might want to check out the Gibbs Phenomenon (https://en.wikipedia.org/wiki/Gibbs_phenomenon) from Fourier analysis and 'ringing artifacts' occurring in signal processing in general.

Answer 4

Your target value is only binary, i.e., takes on two values: 50 and 100. If you rescaled these values, you could recode 50 to 0 and 100 to 1. Then use the logistic activation function on the output-side which will yield predicted values in the range [0,1].

Firstly, however:

-You said nothing about the scale (range, min, max) of your input values. The basic "vanilla-flavored" feed-forward backpropagation neural network likes all input values for all features to be in the range [-1,1], [0,1] based on normalization, or [~ -5, ~5] based on mean-zero standardization.

-You also said nothing about how you are randomizing the connection weights before training. A simple rule of thumb is to use random values in the range [-0.5,0.5].

-You stated nothing about whether you're using cross-validation, leave-one-out, etc.

-When using a simple neural network for function approximation (not classification), sometimes the solution will break-down (not converge) if you don't have uniformly-spaced input features values over the range of their values. For example, if the range of a continuous feature is [10,400] and you feed the net values mostly like 20, 30, 230, 380, the network won't learn the data. This is why Latin hypercube sampling (LHS) is commonly invoked when using a neural network for function approximation.

Your problem is really a classification problem, since y=f(x) takes on discrete values (50,100) and not values which are continuously scaled.

Answer 5

• I think you can try "Mixture of Experts" (MoE) which will help solving a problem by dividing it into smaller, bite-sized pieces, like the brain does

• The two components of MoE:

  • The experts: Instead of just one network, we have several mini-networks called "experts," each good at specific parts of the problem.

  • The gate: Another network, the "gate," decides how much each expert contributes to the final prediction.

-> The final prediction is then the weighted summation of expert predictions