43
$\begingroup$

Can anyone explain what maxout units in a neural network do? How do they perform and how do they differ from conventional units?

I tried to read the 2013 "Maxout Network" paper by Goodfellow et al. (from Professor Yoshua Bengio's group), but I don't quite get it.

|cite|improve this question
$\endgroup$
26
$\begingroup$

A maxout layer is simply a layer where the activation function is the max of the inputs. As stated in the paper, even an MLP with 2 maxout units can approximate any function. They give a couple of reasons as to why maxout may be performing well, but the main reason they give is the following --

Dropout can be thought of as a form of model averaging in which a random subnetwork is trained at every iteration and in the end the weights of the different such random networks are averaged. Since one cannot average the weights explicitly, an approximation is used. This approximation is exact for for a linear network
In maxout, they do not drop the inputs to the maxout layer. Thus the identity of the input outputting the max value for a data point remains unchanged. Thus the dropout only happens in the linear part of the MLP but one can still approximate any function because of the maxout layer.
As the dropout happens in the linear part only, they conjecture that this leads to more efficient model averaging as the averaging approximation is exact for linear networks.

Their code is available here.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thanks! I understand how to do maxout now. I have some questions. (1) So you don't do dropout before maxout out according to what you wrote "In maxout, they do not drop the inputs to the maxout layer. "? (2) What does it mean by the linear part of MLP? "Thus the dropout only happens in the linear part of the MLP but one can still approximate any function because of the maxout layer. " (3) Why better averaging? "As the dropout happens in the linear part only, they conjecture that this leads to more efficient model averaging as the averaging approximation is exact for linear networks." $\endgroup$ – RockTheStar Dec 19 '14 at 18:53
  • 2
    $\begingroup$ (1) So from what I understand, a maxout layer node is $\max (z_i)$ and $z_i = \sum_k x_k$. So the input nodes to the maxout layer themselves are linear combinations of nodes in layers deeper down. Dropout is not done for the maxout layer or for the z layer but it is done for the layers deeper than that. (2) By linear part, it means hidden layers that only have linear activation functions (3) better averaging as for linear networks, the averaging scheme used by dropout is exact. In general NN are not linear but with maxout, they make most of NN linear and only do dropout on the linear part. $\endgroup$ – Opt Dec 20 '14 at 22:46
  • $\begingroup$ Thanks! For (3) what does it mean by "exact", and why NN becomes linear with maxout? Thank you very much for your help. $\endgroup$ – RockTheStar Jan 8 '15 at 18:12
  • 1
    $\begingroup$ Nah maxout is a network architecture. Dropout is a regularization technique $\endgroup$ – Opt Mar 14 '16 at 6:10
  • 1
    $\begingroup$ -1 This is not a correct definition of Goodfellow's maxout activation function. The answer by @toussaint-louverture gets it right. stats.stackexchange.com/a/298705/20587 $\endgroup$ – Trisoloriansunscreen Jan 19 '18 at 21:44
16
$\begingroup$

A maxout unit can learn a piecewise linear, convex function with up to k pieces. 1

So when k is 2, you can implement the ReLU, absolute ReLU, leaky ReLU, etc., or it can learn to implement a new function. If k is let's say 10, you can even approximately learn the convex function.

When k is 2:

the Maxout neuron computes the function $\max(w_1^Tx+b_1, w_2^Tx + b_2)$. Both ReLU and Leaky ReLU are a special case of this form (for example, for ReLU we have $w_1, b_1 = 0$). The Maxout neuron therefore enjoys all the benefits of a ReLU unit (linear regime of operation, no saturation) and does not have its drawbacks (dying ReLU).

However, unlike the ReLU neurons it doubles the number of parameters for every single neuron, leading to a high total number of parameters. 2

You can read the details here:
1. DL book
2. http://cs231n.github.io/neural-networks-1

|cite|improve this answer
$\endgroup$
  • $\begingroup$ It doubles only number of bias parameters, not all weights. $\endgroup$ – hans Feb 26 '18 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.