The exact answer is going to depend greatly on the type of network, the inputs, how it's trained....

In some particular limits (network becoming infinitely wide, initialized near zero, trained via SGD, no batch normalization, ...), the "neural tangent kernel" regime has an answer here: activations are distributed according to a particular Gaussian process, and their derivatives are too. (This means the final result corresponds to kernel ridge regression / Gaussian process regression with a particular kernel.) See e.g.

- Appendix 2 of [Jacot et al. (2018), _Neural Tangent Kernel: Convergence and Generalization in Neural Networks_](http://arxiv.org/abs/1806.07572), or 
- Appendix D of [Arora et al. (2019), _On Exact Computation with an Infinitely Wide Neural Net_](http://arxiv.org/abs/1904.11955).