This question has two parts:
- Is it possible to formally define the set of all activation functions? The vector space of all functions might suffice, but I'm not sure how to do 2 with this, so something else might be better.
- Given this set, sample evenly from it so that as the size of your samples grow, your sample begins to represent this entire set's distribution more and more?
The hope here is that by training a neural network to solve some problem multiple times with each random sample you have chosen, you will begin to see how this neural network will perform on average given any activation function, which would help you see how easily learnable that data is in general.