Creating thinned models in during Dropout process 
Applying dropout to a neural network amounts to sampling a “thinned” network from it. The thinned network consists of all the units that survived dropout. A neural net with $n$ units can be seen as a collection of $2^n$ possible thinned neural networks.

Source:
Dropout:  A Simple Way to Prevent Neural Networks fromOverfitting, pg. 1931.
How are we getting these $2^n$ models?
 A: The statement is a bit oversimplifying but the idea is that assuming we have $n$ nodes and each of these nodes might be "dropped", we have $2^n$ possible thinned neural networks. Obviously dropping out an entire layer would alter the whole structure of the network but the idea is straightforward: we ignore the activation/information from certain randomly selected neurons and thus encourage redundancy learning and discourage over-fitting  on very specific features. 
The same idea has also been employed in Gradient Boosting Machines where instead of "ignoring neurons" we "ignore trees" at random (see Rashmi & Gilad-Bachrach (2015) DART: Dropouts meet Multiple Additive Regression Trees on that matter).
Minor edit: I just saw Djib2011's answer. (+1) He/she specifically shows why the statement is somewhat over-simplifying. If we assume that we can drop any (or all, or none) of the neurons we have $2^n$ possible networks.
A: assume we have n neurons and each neuron has the probability to be disabled.
situation 0: zero neuron remains, n neurons are disabled, C(n,0)
situation 1: only one neuron remains, n-1 neurons are disabled, C(n,1)
situation 2: only two neurons remain, n-2 neurons are disabled, C(n,2)
.
.
.
situation n: n neurons remain, 0 neurons are disabled, C(n,n)
so C(n,0)+C(n,1)+C(n,2)+...+ C(n,n)=2^n
A: I too haven't understood their reasoning, I always assumed it was a typo or something...
The way I see it we if we have $n$ hidden units in a Neural Network with a single hidden layer and we apply dropout keeping $r$ of those, we'll have:
$$
\frac{n!}{r! \cdot (n-r)!}
$$
possible combinations (not $2^n$ as the authors state).

Example:
Assume a simple fully connected neural network with a single hidden layer with 4 neurons. This means the hidden layer will have 4 outputs $h_1, h_2, h_3, h_4$.
Now, you want to apply dropout to this layer with a 0.5 probability (i.e. half of the outputs will be dropped).
Since 2 out of the 4 outputs will be dropped, at each training iteration we'll have one of the following possibilities:


*

*$h_1, h_2$

*$h_1, h_3$

*$h_1, h_4$

*$h_2, h_3$

*$h_2, h_4$

*$h_3, h_4$
or by applying the formula:
$$
\frac{4!}{2! \cdot (4-2)!} = \frac{24}{2 \cdot 2} = 6
$$
A: This is quite simple. It can be thought as a task of getting the number of subsets from one set. 
