I've been trying to figure out what exactly it meant by the "Maxout" activation function in neural networks. There is this question, this paper, and even in the Deep Learning book by Bengio et al., except with just little bit of information and a big TODO next to it.
I will be using the notation described here for clarity. I just don't want to retype it up and cause question bloat. Briefly, $a^i_j=\sigma(z^i_j)=\sigma(\sum\limits_k a^{i-1}_kw^i_{jk}+b^i_j)$, in other words, a neuron has a single bias, a single weight for each input, and then it sums the inputs times the weights, then adds the bias and applies the activation function to get the output (aka activation) value.
So far I know that Maxout is an activation function that "outputs the max of it's inputs". What does that mean? Here are some ideas that I could interpret from that:
- $a^i_j=\max\limits_k (a^{i-1}_k)$, also known as max-pooling.
- $a^i_j=\max\limits_k (a^{i-1}_kw^i_{jk})+b^i_j$, simply replacing the sum that is normally done with a max.
- $a^i_j=\max\limits_k (a^{i-1}_kw^i_{jk}+b^i_{jk})$, where each neuron now has one bias value for each input, instead of a single bias value applied after summing all inputs. This would make backpropagation different, but still possible.
- Each $z^i_j$ is computed as normal, and each neuron has a single bias and a weight for each input. However, similar to softmax ($a^i_j = \frac{\exp(z^i_j)}{\sum\limits_k \exp(z^i_k)}$), this takes the maximum of all $z$'s in it's current layer. Formally, $a^i_j=\max\limits_k z^i_k$.
Are any of these correct? Or is it something different?