Neural Network Ensemble Averaging

Question

I have the following question:

Assume I am given a function $f(x)$ that I want to do neural network regression on (for some subset of $x$), and further assume that I can get as many $(x,f(x))$ pairs as I want for training. Given that I train $N$ neural networks ($NN_i(x)$, $i=1,...,N$) independently using random initial weights and MSE as my loss, am I guaranteed that

as $\underset{N \to \infty}{lim} ~ |f(x) - \frac{1}{N}\sum_i^N{NN_i(x)}| = 0 ~ ~ \forall x~ ~$?

If yes, why?

If not, what assumptions are necessary for the statement to hold?

Thank you, Mr.Red

EDIT:

Maybe to make the answer more meaningful, let's assume f(x) is continuous and the nn is single layered with as many neurons as desired and training is done via some gradient descent method.

Answer 1

If not, what assumptions are necessary for the statement to hold?

One necessary assumption is that the neural networks have the capacity to learn $f$.

Answer 2

Firstly, even a single hidden layer neural network with as many hidden units as needed, can approximate any function (the so called universal approximation property of neural networks http://deeplearning.cs.cmu.edu/pdfs/Kornick_et_al.pdf). However, it is more difficult (if not impossible, depending on the complexity of the function) to train such a large network. Deep learning tries to ease (not solve) this by adding more layers, i.e., increasing the capacity to learn more complex functions without increasing the complexity during learning. So, for all practical purposes, let's assume that each neural network has a finite capacity (which is a function of number of parameters in the model).

That being said, what you have described above is model averaging with equal weights to each model, called as Committee methods. A more useful way of doing model averaging is weighted averaging using BIC Elements of Statistical Learning, Sec. 8.8. It can shown (same reference)

the full regression has smaller error than any single model, so combining models never makes things worse, at the population level.

Note that this model averaging can be on any regression, need not be neural networks.