Universal Approximation: how does a neural network handle a ratio of inputs

Question

Related questions/background info:

• Universal Approximation Theorem — Neural Networks
• Does the universal approximation theorem for neural networks hold for any activation function?

A universal approximator is a device or technique that is capable of approximating an input to any desired accuracy, i.e. such that the error $\epsilon$ can be made as small as desired. In the context of neural networks, there are a number of "universal approximation theorems" since, in the words of Kratsios

The universal approximation property of various machine learning models is currently only understood on a case-by-case basis

The singular term Universal Approximation Theorem therefore refers to

results that establish the density of an algorithmically generated class of functions within a given function space of interest

To define the scope of this question in terms of such spaces with sufficient mathematical rigour to satisfy the demands of all is beyond my ability, so for the purposes of this question, answers may refer to any known Universal Approximation Theorem whose scope does not exclude the specific example given. Caveats and constraints arising from particulars of the function (e.g. absence of divergence, Lebesgue integrability, etc.) may be added as required.

There are many nice explanations of how neural networks can be universal approximators (see e.g. A visual proof that neural nets can compute any function) in the context of simple polynomials, but how does it work when, for example, the function to be approximated involves a ratio, e.g. (arbitrary example)

$$ f(x, y, z) = {x^3 * y}/z $$

Question Given that most neural networks are constructed from nodes that multiply and add (via the weights and biases of the matrices), how can such a system be a universal approximator for functions that include other mathematical forms such as ratios?

Then, given any suitable function (with the space of the UAT being discussed), are there any results that place lower bounds on network size (layers, breadth, depth, etc.), number of training examples required, etc. in order to achieve error < some specified $\epsilon$.

Answer 1

Given that most neural networks are constructed from nodes that multiply and add (via the weights and biases of the matrics), how can such a system be a universal approximator for functions that include other mathematical forms such as ratios?

I think what you’re asking for is a demonstration of why rational functions can be modeled as neural networks on an interval that does not include a singularity. Approximating a rational function such as $f(z)=z^{-1}$ in an interval “away from” the singularity can be done using piecewise linear functions. The simplest approximation uses 1 line, and has a high error; but we are “away from” the singularity, so the error is finite. We can reduce the high error by adding more linear basis functions, breaking the curve into smaller lines. In terms of neural networks, ReLU activation functions yield piecewise linear prediction functions.

The main idea, breaking a curve into lines using ReLU, is outlined in How does the Rectified Linear Unit (ReLU) activation function produce non-linear interaction of its inputs? where the function being approximated is a simple quadratic.

This construction shows how multiplication and addition can be composed with a common neural network activation to create an approximation to a rational function on an interval omitting the singularity.

Answer 2

The usual universal approximation theorem involves a continuous function between spaces. If your ratio includes a zero-denominator point, either a $0/0$ point discontinuity or a function that shoots up to infinity, then the function being approximated is discontinuous, and the universal approximation theorem does not apply.

If you restrict your domain to where the function is continuous, then there is no issue. You can approximate something like $f(x) = 1/x$ on $[1,2]$.