Given any number of points, there are an infinite number of functions these points can be samples from. For $n$ points, any polynomial of degree $> n$ can be used to generate these points.
If there are an infinite number of functions, how does the net know which one to approximate? I know that ideal NN's are universal function approximators, but obviously real ones have limitations based on their architecture. But architecture limitations aside, how does the net "choose" which function to approximate? For any error function, there are infinite functions (again, architecture limitations aside) that would give you zero training error.
I know that not all error functions are convex, and they often have multiple minimas. But I'm pretty sure someone showed a while back that all the minimas tend to have relatively the same performance.
Do neural networks implicitely maybe make some sort of assumption on the function it tries to approximate? Maybe some sort of minimization of complexity of the approximated function? I don't see how data points are sufficient for learning...