This question has two parts:

  1. Is it possible to formally define the set of all activation functions? The vector space of all functions might suffice, but I'm not sure how to do 2 with this, so something else might be better.
  2. Given this set, sample evenly from it so that as the size of your samples grow, your sample begins to represent this entire set's distribution more and more?

The hope here is that by training a neural network to solve some problem multiple times with each random sample you have chosen, you will begin to see how this neural network will perform on average given any activation function, which would help you see how easily learnable that data is in general.


Here is where you'll run into a lot of problems. The set of all functions $f : R \to R$ is very, very large and in no way countable (I don't even believe it's measurable). What you can settle for is a something like a mixing of activation functions:

$$f(x, p_1, p_2, ..., p_n) = p_1 f_1(x) + p_2 f_2(x) + ... + p_n f_n(x)$$

$f_i$ are specific activation functions which all share a domain and range and $p_i$ are scalar values. This will not give you all functions, but then you might be able to aggregate your average over all p.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.