How to extract the function being approximated by a neural network? We all know that in general, a neural network takes in a set of training examples having the form $\{x, f(x)\}$ and it aims to approximate the function $f$ thereby "classifying" $x$ to its correct output.
This not only applies in function approximation but in virtually any domain. Classification problems are essentially function approximation problems where the function classifies an input $x$ appropriately.
This means that in general, for any neural network problem the aim is always to approximate the target function $f(x)$. In practice, we rarely care what $f(x)$ really is, so long as the neural network does its job correctly. For example, if we train a neural network to correctly classify between an apple and an orange given a feature vector $x$, we don't really think about the actual function $f$ being approximated in the process. Namely, the function which essentially does all the magic of classifying between an apple and an orange.
My question is, are there ways to actually extract this explicit, $f$ approximate; $\hat{f}$? Both in practice and in theory?
 A: In theory: The structure of the network -- how many layers, how many nodes in each layer and what the activation function is -- gives you the general functional form of the network. Add the estimated weights to get the particular estimated function $\hat f$. All of this should in principle be known/available to the "user" of the network. 
In applications: Software that implements the neural network will often take the structure (number of layers, number of nodes and activation function) as arguments and will output the estimated weights if requested.
Thus the explicit function $\hat f$ (the network's approximation of the underlying true function $f$) can be extracted, both in theory and in practice. However, I do not see how the true function $f$ could be extracted, and I think in general this may not be possible.
A: Let's denote $f$ the true underlying function and $\hat f$ the function that your machine learning algorithm converges too ($\hat f$ belongs to a family of parametrized functions $F$).
For simplicity, let's also assume that $f$ can be expressed analytically and that $f$ is deterministic.

My question is, are there ways to actually extract this explicit
  function f? Both in practice and in theory?

I assume that by "practice", you mean with machine learning (using experimental data) and by "theory", you mean modelling mathematically without machine learning (without data).
In practice, if you have enough data and if $F$ contains $f$, then it should be possible to obtain $\hat f$ = $f$ with an appropriate machine learning methodology. 
Theoretically, you may try to model $f$ with physical laws (or other modelling laws). For example is $f(p,s)$ models the time it takes for an object of shape s and weight p to fall from the top of the Eiffel tower, you can use classical mechanics (assuming they are true in the scope/scale of $f$) to model $f$.
For apple and oranges, $f$ is subjective to a particular person (given an ambiguous picture, two persons may disagree). So let's consider your $f$. $f$ is then defined by your brain! So if we assume that there exists an analytical expression of $f$, here are the two ways to find it:


*

*"in practice" (with machine learning): choose F sufficiently large to model the brain (the brain has more than 80 billions of neurons...), build a big enough dataset and choose a good machine learning algorithm. Ideally the dataset should contain all the possible images of oranges and apples. Then train until you get a null training error AND a null generalisation error.

*"in theory" (modelling): model the network of biological neurons of your brain. The problem is that we do not yet understand how the brain works.


To recap, you can usually find $f$ but it is really hard in both cases:


*

*in practice: you need a good F, enough data and a good enough ML algorithm.

*in theory: you need to know all the "physical laws" and be sure they are correct in the scope of the function $f$.

