Universal approximation theorem - representing versus learning functions

Question

In Goodfellow et al.'s Deep Learning, the authors write, "the universal approximation theorem means that regardless of what we are trying to learn, we know that a large MLP will be able to represent this function. We are not guaranteed, however, that the training algorithm will be able to learn that function" (193). What is the difference between "representing" and "learning" a function? For example, if the function is a polynomial of order $n$, does the theorem state that a neural network can learn polynomials, but the parameters of the specific polynomial of interest may not be learnable based on the number of neurons / number of data points?

Answer 1

Consider nine data points that has been generated from a tenth degree polynomial. Even if the data was generated with no noise, i.e. the y-coordinates are the exact function values of the polynomial, no algorithm can correctly identify the true underlying polynomial - the data just does not have enough information content to do so.

This is true even if the shapes your model can possibly capture include all tenth degree polynomials. I.e. even if you fit a tenth degree polynomial to your data along with some regularization strategy to deal with the over-specification.

In practice, the complexity of your model specification needs to depend on the quantity and quality of the data you have available. If you have nine data points, fitting a tenth degree polynomial is a bad idea, even if you know the correct answer is a tenth degree polynomial. The data you have just does not allow you to identify a shape of that complexity, you are better off greatly under specifying your model in cases like this.

The functions your model can represent are all the possible functional forms your final model can assume. The functions your model can learn are all those that can realistically be confidently determined from your data. Note that the first is only dependent on how you specify your model, but the second is dependent on both how you specify your model, and the data you have available.