How are groups created in maxout units when dividing the set of inputs 𝑧 into groups of 𝑘 values?

Question

I don't get $G^(i)$the set of indices into the inputs for group $i$, $\{(i −1)k+ 1, \ldots , ik\}$ when creating a maxout units/function, these thing that outputs the maximum element of groups: $$g(z)_i= \max_{[j∈G^{(i)}]}z_j$$

Maxout units(Goodfellow et al., 2013a) generalize rectified linear units. Instead of applying an element-wise function $g(z)$, maxout units divide $z$ into groups of $k$ values. Each maxout unit then outputs the maximum element of one of these groups:

$$g(z)_i= \max_{[j∈G^{(i)}]}z_j$$

where $G^{(i)}$is the set of indices into the inputs for group $i$,$\{(i −1)k+ 1, \ldots , ik\}$.

This provides a way of learning a piecewise linear function that responds to multiple directions in the input $x$ space. A maxout unit can learn a piecewise linear, convex function with up to $k$ pieces. Maxout units can thus be seen as learning the activation function itself rather than just the relationship between units. With large enough $k$, a maxout unit can learn to approximate any convex function with arbitrary fidelity. In particular, a maxout layer with two pieces can learn to implement the same function of the input $x$ as a traditional layer using the rectified linear activation function, the absolute value rectification function, or the leaky or parametric ReLU, or it can learn to implement a totally different function altogether. The maxout layer will, of course, be parametrized differently from any of these other layer types, so the learning dynamics will be different even in the cases where maxout learns to implement the same function of $x$ as one of the other layer types.

Deep Learning, Bengio and al. page 190

I tried to work it out with a concrete example:

Let's take a concrete example with k = 3. In this case, the groups would be as follows:

$$G^{(1)} = {(1 - 1)3 + 1, \ldots, 1 * 3} = \{1, 2, 3\}$$ $$G^{(2)} = {(2 - 1)3 + 1, \ldots, 2 * 3} = \{4, 5, 6\}$$ $$G^{(3)} = {(3 - 1)3 + 1, \ldots, 3 * 3} = \{7, 8, 9\}$$ And so on...

So, in this example, if we have 9 inputs, we'd have 3 maxout units, and each maxout unit would use 3 inputs to calculate its output, choosing the maximum from these 3 values.

Suppose we want to model the cost of producing a certain number of units of military weapons as a function of the quantity produced. We can use a piecewise linear function for this.

So let's imagine that the production cost is defined by a convex function with three pieces (k = 3):

• For the first 100 units produced, the cost is \$10,000 per unit.
• For 101 to 200 units, the cost is \$8,000 per unit.
• For 201 units and above, the cost is \$7,000 per unit.

So here, I guess each maxout unit would take as input the quantity produced, and each group $G^(i)$ would be associated with one of the pieces of the function.

For example, if we wanted to calculate the cost of production for 150 units, I'm not sure but I guess we'd have :

$G^{(1)} = \{1, 2, 3\}$ corresponding to the first part of the function (\$10,000 per unit). $G^{(2)} = \{4, 5, 6\}$ corresponding to the second part of the function (\$8,000 per unit).

I don't get what are $\{1, 2, 3\}$ and $\{4, 5, 6\}$ standing for, concretely

If I apply the maxout function for each group, I guess it means that maxout unit 1 will select the maximum of the values associated with $G^{(1)}$, and maxout unit 2 will select the maximum of the values associated with $G^{(2)}$, and so on ... but then it doesn't make any sens

Indeed, for 150 units, I guess maxout 1 would give a value of \$10,000 per unit, and maxout 2 would give a value of \$10,000 per unit, ... I don't know how I am expected to calculate the total cost of production then ...

I am a slow but passionate learner in math and deep learning. Don't hesitate to explain it to me as if I was a teenager.

Answer 1

Your example function does not really have multiple inputs in the same sense as max-out functions have multiple inputs. Your example function is in essence just a scalar function ($\mathbb{R} \to \mathbb{R}$). The max-out non-linearity, on the other hand, is actually a vector function ($\mathbb{R}^N \to \mathbb{R}^{N/k}$).

Take the following six-dimensional vector (i.e. $N = 6$) as an example input: $\boldsymbol{x} = (1, 3, -2, -4, -1, 3)$. Applying the max-out non-linearity with $k = 2$ to $\boldsymbol{x}$ would result in $\operatorname{max-out}(\boldsymbol{x}) = (3, -2, 3) = \big(\max(1, 3), \max(-2, -4), \max(-1, 3)\big).$ Similarly, using max-out non-linearity with $k = 3$, we would obtain $\operatorname{max-out}(\boldsymbol{x}) = (3, 3).$

If $k$ does not divide $N$, this just does not work. The only thing to make things work is to either crop the input vector (ignore the last few outputs) or pad the input vector with $p$ zeros such that $k$ divides $N + p$.

Finally, the max-out non-linearity should not be thought of as a function that models something useful. It was conceived as an activation function for hidden representations in neural networks. Typically, these hidden representations are hard to interpret. Therefore, it is not really important that these activation functions are easily interpretable. The only thing we care about is that the non-linearity is effectively non-linear.