Ability of neural network to learn a random neural network In experimenting with the power of basic neural networks to learn particular functions, I was wondering whether a neural network can learn a random neural network. That is, if we take a neural network $N_0$ of some fixed shape, and randomly initialize the parameters, and then take another neural network $N_1$ of the same shape and train $N_1$ to produce the function given by $N_0$, what happens?
I feel like this is a natural question, and predict that there are others who have tried to do the same. I'm finding it difficult to search for relevant results, though.
Can anyone tell me whether this has been tried before, and where I can find the experimenters' observations?
 A: I'm not aware of any studies to this effect. As Sycorax pointed out in the comments, neural networks are universal function approximators. In theory you could learn to approximate any neural network $N_0$ with another neural network $N_1$. But as Bridgeburners points out, the more complicated $N_0$ the less likely $N_1$ is to learn it correctly. For this reason, I don't think anyone has found it worthwhile to study in general.
If you're interested in reverse-engineering neural networks, then there are some results that might help you:


*

*Tramèr, F., et al. (2016). Stealing machine learning models via prediction APIs. 25th Usenix Security Symposium.


*

*https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer

*https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_tramer.pdf

*https://qz.com/786219/stealing-an-ai-algorithm-and-its-underlying-data-is-a-high-school-level-exercise/


*Oh, S., et al. (2018). Towards reverse-engineering black-box neural networks. ICLR 2018.


*

*https://arxiv.org/abs/1711.01768

*https://arxiv.org/pdf/1711.01768.pdf

*https://openreview.net/forum?id=BydjJte0-

*https://openreview.net/pdf?id=BydjJte0-

*https://iclr.cc/Conferences/2018/Schedule?showEvent=243

*https://github.com/coallaoh/WhitenBlackBox
