# Exact definition of Maxout

I've been trying to figure out what exactly it meant by the "Maxout" activation function in neural networks. There is this question, this paper, and even in the Deep Learning book by Bengio et al., except with just little bit of information and a big TODO next to it.

I will be using the notation described here for clarity. I just don't want to retype it up and cause question bloat. Briefly, $a^i_j=\sigma(z^i_j)=\sigma(\sum\limits_k a^{i-1}_kw^i_{jk}+b^i_j)$, in other words, a neuron has a single bias, a single weight for each input, and then it sums the inputs times the weights, then adds the bias and applies the activation function to get the output (aka activation) value.

So far I know that Maxout is an activation function that "outputs the max of it's inputs". What does that mean? Here are some ideas that I could interpret from that:

1. $a^i_j=\max\limits_k (a^{i-1}_k)$, also known as max-pooling.
2. $a^i_j=\max\limits_k (a^{i-1}_kw^i_{jk})+b^i_j$, simply replacing the sum that is normally done with a max.
3. $a^i_j=\max\limits_k (a^{i-1}_kw^i_{jk}+b^i_{jk})$, where each neuron now has one bias value for each input, instead of a single bias value applied after summing all inputs. This would make backpropagation different, but still possible.
4. Each $z^i_j$ is computed as normal, and each neuron has a single bias and a weight for each input. However, similar to softmax ($a^i_j = \frac{\exp(z^i_j)}{\sum\limits_k \exp(z^i_k)}$), this takes the maximum of all $z$'s in it's current layer. Formally, $a^i_j=\max\limits_k z^i_k$.

Are any of these correct? Or is it something different?

## 1 Answer

None of the above; maxout networks don't follow the architecture you assumed.

From the beginning of the "description of maxout" section in the paper you linked, which defined maxout:

Given an input $x \in \mathbb{R}^d$ ($x$ may be $v$, or may be a hidden layer’s state), a maxout hidden layer implements the function

$$h_i = \max_{j \in [1, k]} z_{ij}$$

where $z_{ij} = x^T W_{ij} + b_{ij}$, and $W \in \mathbb{R}^{d \times m \times k}$ and $b ∈ R^{m \times k}$ are learned parameters.

So, each unit of the $m$ units has $k$ different affine combinations of the previous layer, and outputs the max of those $k$ affine functions. Imagine each layer being conected to the previous layer with $k$ different-colored connections, and taking the max of the colors.

Alternatively, you can think of a maxout unit as actually being two layers: each of the previous layer's units is connected to each of $k$ units with the identity activation function, and then a single unit connects those $k$ linear units with a max-pooling activation.

This means that the unit, viewed as a function from $\mathbb R^d$ to $\mathbb R$, is the piecewise max of affine functions. The paper's Figure 1 gives some examples of different functions it might look like: Each of the dashed lines represents a $W^T x + b$. You can represent any convex function in this way, which is pretty nice.

• So since $z_{i j} \in \mathbb{R}^{m \times k}$, that means that $z_{i 1},z_{i 2}, ... \in \mathbb{R}^m$. How does one take the max of this? Is it magnitude of the vectors? – Phylliida Jun 4 '15 at 12:58
• @DanielleEnsign The indexing is a little bit nonstandard here, but each $W_{ij} \in \mathbb R ^d$, $x \in \mathbb R ^d$, $b_{ij} \in \mathbb R$ so $z_{ij} \in \mathbb R$. It's a normal, scalar max. – Danica Jun 4 '15 at 18:04
• Ah I finally get it now, thanks. Basically each neuron is composed of lots of "subneurons" that take in the input to that neuron, have their own weights and biases, and output via the identity activation function. Then the output to that neuron is the max of all it's sub-neuron's outputs. – Phylliida Jun 4 '15 at 23:18
• Would it be accurate to say that you can model this using k multiple connections between each pair of connected neurons (rather than a single connection as is normally done), calculating k activations for each pair and then selecting the top one as the winner? Or is it sometimes necessary to use separate biases for each sub-connection, thereby making it necessary to model each connection as if it belong to a different sub-neuron? – SQLServerSteve Jul 29 '16 at 2:42
• @SQLServerSteve Each of the $k$ connections can have different biases (this is necessary e.g. for the "quadratic" in the picture above). But you can still think about it in the way you described, you just also have to also add in a bias for each of the connections. Sometimes people talk about an imaginary "bias unit" that always outputs 1; in that case, your model works fine as long as the bias unit also has multiple connections. – Danica Jul 29 '16 at 2:49