What is maxout in neural network? Can anyone explain what maxout units in a neural network do? How do they perform and how do they differ from conventional units?
I tried to read the 2013 "Maxout Network" paper by Goodfellow et al. (from Professor Yoshua Bengio's group), but I don't quite get it.
 A: A maxout layer is simply a layer where the activation function is the max of the inputs. As stated in the paper, even an MLP with 2 maxout units can approximate any function. They give a couple of reasons as to why maxout may be performing well, but the main reason they give is the following --
Dropout can be thought of as a form of model averaging in which a random subnetwork is trained at every iteration and in the end the weights of the different random networks are averaged. Since one cannot average the weights explicitly, an approximation is used. This approximation is exact for a linear network 
In maxout, they do not drop the inputs to the maxout layer. Thus the identity of the input outputting the max value for a data point remains unchanged. Thus the dropout only happens in the linear part of the MLP but one can still approximate any function because of the maxout layer. 
As the dropout happens in the linear part only, they conjecture that this leads to more efficient model averaging as the averaging approximation is exact for linear networks.
Their code is available here.
A: 
A maxout unit can learn a piecewise linear, convex function with up to k pieces. 1

So when k is 2, you can implement the ReLU, absolute ReLU, leaky ReLU, etc., or it can learn to implement a new function. If k is let's say 10, you can even approximately learn the convex function.
When k is 2:

the Maxout neuron computes the function $\max(w_1^Tx+b_1, w_2^Tx + b_2)$. Both ReLU and Leaky ReLU are a special case of this form (for example, for ReLU we have $w_1, b_1 = 0$). The Maxout neuron therefore enjoys all the benefits of a ReLU unit (linear regime of operation, no saturation) and does not have its drawbacks (dying ReLU). 
However, unlike the ReLU neurons it doubles the number of parameters for every single neuron, leading to a high total number of parameters. 2


You can read the details here:
1. DL book
2. http://cs231n.github.io/neural-networks-1

