9
$\begingroup$

We all know that in general, a neural network takes in a set of training examples having the form $\{x, f(x)\}$ and it aims to approximate the function $f$ thereby "classifying" $x$ to its correct output.

This not only applies in function approximation but in virtually any domain. Classification problems are essentially function approximation problems where the function classifies an input $x$ appropriately.

This means that in general, for any neural network problem the aim is always to approximate the target function $f(x)$. In practice, we rarely care what $f(x)$ really is, so long as the neural network does its job correctly. For example, if we train a neural network to correctly classify between an apple and an orange given a feature vector $x$, we don't really think about the actual function $f$ being approximated in the process. Namely, the function which essentially does all the magic of classifying between an apple and an orange.

My question is, are there ways to actually extract this explicit, $f$ approximate; $\hat{f}$? Both in practice and in theory?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ In the last paragraph, shouldn't you use $\hat f$ in place of $f$? It seems to me that there you are referring to the approximated function as given by the trained network, not the original "true" function. $\endgroup$
    – Richard Hardy
    Commented Apr 11, 2015 at 19:03
  • $\begingroup$ The question is not very clear and by looking at the answers it is not related to neural networks. Are you asking whether real f can be recovered or whether NN representation of it can be written on paper like a math formula? $\endgroup$
    – Cagdas Ozgenc
    Commented Apr 12, 2015 at 13:33
  • $\begingroup$ I'm asking whether the real $f$ can be recovered. Technically since we can never really reach $f$ but only its approximate $\hat{f}$ it would makes more sense to ask if $\hat{f}$ can be found such that its explicit functional representation is more descriptive of the process in question (like $f$ would be) rather than a summation of activation functions etc. $\endgroup$
    – mesllo
    Commented Apr 12, 2015 at 14:03

2 Answers 2

Reset to default
5
$\begingroup$

Let's denote $f$ the true underlying function and $\hat f$ the function that your machine learning algorithm converges too ($\hat f$ belongs to a family of parametrized functions $F$). For simplicity, let's also assume that $f$ can be expressed analytically and that $f$ is deterministic.

My question is, are there ways to actually extract this explicit function f? Both in practice and in theory?

I assume that by "practice", you mean with machine learning (using experimental data) and by "theory", you mean modelling mathematically without machine learning (without data).

In practice, if you have enough data and if $F$ contains $f$, then it should be possible to obtain $\hat f$ = $f$ with an appropriate machine learning methodology.

Theoretically, you may try to model $f$ with physical laws (or other modelling laws). For example is $f(p,s)$ models the time it takes for an object of shape s and weight p to fall from the top of the Eiffel tower, you can use classical mechanics (assuming they are true in the scope/scale of $f$) to model $f$.

For apple and oranges, $f$ is subjective to a particular person (given an ambiguous picture, two persons may disagree). So let's consider your $f$. $f$ is then defined by your brain! So if we assume that there exists an analytical expression of $f$, here are the two ways to find it:

  • "in practice" (with machine learning): choose F sufficiently large to model the brain (the brain has more than 80 billions of neurons...), build a big enough dataset and choose a good machine learning algorithm. Ideally the dataset should contain all the possible images of oranges and apples. Then train until you get a null training error AND a null generalisation error.
  • "in theory" (modelling): model the network of biological neurons of your brain. The problem is that we do not yet understand how the brain works.

To recap, you can usually find $f$ but it is really hard in both cases:

  • in practice: you need a good F, enough data and a good enough ML algorithm.
  • in theory: you need to know all the "physical laws" and be sure they are correct in the scope of the function $f$.
Improve this answer
$\endgroup$
1
  • $\begingroup$ In fact your theoretical explanation is exactly the function $f$ that I want to find but using neural networks instead of manually deriving it. Indeed, this is probably impossible using neural networks and only achievable by analysing the process itself and using physical laws (or other laws if it does not involve physics) to derive an explicit formulation. Indeed, this is so because neural networks only express $f$ (or its approximation at least) with a superposition of activation functions, so that is probably the closest explicit representation we could have by using neural networks. $\endgroup$
    – mesllo
    Commented Apr 12, 2015 at 12:36
6
$\begingroup$

In theory: The structure of the network -- how many layers, how many nodes in each layer and what the activation function is -- gives you the general functional form of the network. Add the estimated weights to get the particular estimated function $\hat f$. All of this should in principle be known/available to the "user" of the network.

In applications: Software that implements the neural network will often take the structure (number of layers, number of nodes and activation function) as arguments and will output the estimated weights if requested.

Thus the explicit function $\hat f$ (the network's approximation of the underlying true function $f$) can be extracted, both in theory and in practice. However, I do not see how the true function $f$ could be extracted, and I think in general this may not be possible.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think you may be right. I guess the only way would be to express the true function $f$ using $\hat{f}$ in the sense that rather than having an explicit expression for $f$ you're using the superpositions of the activation functions in $\hat{f}$ which approximate $f$. Essentially it is still a representation of $f$ but only through the activation functions (by the Universal Approximation Theorem) rather than by the process describing $f$ itself. $\endgroup$
    – mesllo
    Commented Apr 12, 2015 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.