If I have understood this correctly, a result from Hornik et al.'s Universal Approximation of an Unknown Mapping and Its Derivatives Using Multilayer Feedforward Networks essentially states that, if we have a function $F\in C^n$ and a neural network $F_{ANN}$ that approximates $F$, then, if an activation function that is of class $C^n$ is used, then $F_{ANN}$ arbitrarily approximates $F$ and all its derivatives up to order $n$.

($F$ is said to be of class $C^{n}$ if its first through to $n$ derivatives are continuous).

If, for example, we have a neural network with 3 hidden layers, each using an activation function that is of class $C^n$, and an output layer that simply uses the linear (identity) activation function, does the network still approximate $F$ and all of its derivatives up to order $n$?

My thinking is that because the output layer activation is just the linear activation function, the answer is yes. But I am not sure, as the activation function used in the hidden layers (which is of class $C^n$) is not used on the output layer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.