3
$\begingroup$

Related questions/background info:

A universal approximator is a device or technique that is capable of approximating an input to any desired accuracy, i.e. such that the error $\epsilon$ can be made as small as desired. In the context of neural networks, there are a number of "universal approximation theorems" since, in the words of Kratsios

The universal approximation property of various machine learning models is currently only understood on a case-by-case basis

The singular term Universal Approximation Theorem therefore refers to

results that establish the density of an algorithmically generated class of functions within a given function space of interest

To define the scope of this question in terms of such spaces with sufficient mathematical rigour to satisfy the demands of all is beyond my ability, so for the purposes of this question, answers may refer to any known Universal Approximation Theorem whose scope does not exclude the specific example given. Caveats and constraints arising from particulars of the function (e.g. absence of divergence, Lebesgue integrability, etc.) may be added as required.

There are many nice explanations of how neural networks can be universal approximators (see e.g. A visual proof that neural nets can compute any function) in the context of simple polynomials, but how does it work when, for example, the function to be approximated involves a ratio, e.g. (arbitrary example)

$$ f(x, y, z) = {x^3 * y}/z $$

Question Given that most neural networks are constructed from nodes that multiply and add (via the weights and biases of the matrices), how can such a system be a universal approximator for functions that include other mathematical forms such as ratios?

Then, given any suitable function (with the space of the UAT being discussed), are there any results that place lower bounds on network size (layers, breadth, depth, etc.), number of training examples required, etc. in order to achieve error < some specified $\epsilon$.

$\endgroup$
3
  • 1
    $\begingroup$ The $f$ includes a singularity at $z=0$, which excludes it from at least some of the UATs. $\endgroup$
    – Sycorax
    Jun 14 '21 at 15:05
  • $\begingroup$ @sycorax Exclude z=0. I am aware of the varieties and I don't want to go into details of activation functions etc. etc. etc. Consider the question in any way that it is answerable by you and add caveats, conditions as appropriate assuming "reasonable" domain etc. This is a pragmatic question; mathematical niceties will be appreciated but are not the primary focus. Clearly you are able to answer this, so thank you in advance! $\endgroup$ Jun 14 '21 at 15:50
  • $\begingroup$ Thanks for making the edits to your question! The revision allowed me to understand how to answer your question. $\endgroup$
    – Sycorax
    Jun 15 '21 at 13:38
2
$\begingroup$

Given that most neural networks are constructed from nodes that multiply and add (via the weights and biases of the matrics), how can such a system be a universal approximator for functions that include other mathematical forms such as ratios?

I think what you’re asking for is a demonstration of why rational functions can be modeled as neural networks on an interval that does not include a singularity. Approximating a rational function such as $f(z)=z^{-1}$ in an interval “away from” the singularity can be done using piecewise linear functions. The simplest approximation uses 1 line, and has a high error; but we are “away from” the singularity, so the error is finite. We can reduce the high error by adding more linear basis functions, breaking the curve into smaller lines. In terms of neural networks, ReLU activation functions yield piecewise linear prediction functions.

The main idea, breaking a curve into lines using ReLU, is outlined in How does the Rectified Linear Unit (ReLU) activation function produce non-linear interaction of its inputs? where the function being approximated is a simple quadratic.

This construction shows how multiplication and addition can be composed with a common neural network activation to create an approximation to a rational function on an interval omitting the singularity.

$\endgroup$
1
  • $\begingroup$ Yes, that is the conclusion I came to independently. The key point is to avoid the category error: the numerical values of the function are approximated, not the algebraic form. $\endgroup$ Jun 15 '21 at 13:36
1
$\begingroup$

The usual universal approximation theorem involves a continuous function between spaces. If your ratio includes a zero-denominator point, either a $0/0$ point discontinuity or a function that shoots up to infinity, then the function being approximated is discontinuous, and the universal approximation theorem does not apply.

If you restrict your domain to where the function is continuous, then there is no issue. You can approximate something like $f(x) = 1/x$ on $[1,2]$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.