Computation time with respect to Dropout

I've been recently attempting to speed up neural network training (in PyTorch). My question is the following.

Does the computation time of a given feedforward neural network vary based on Dropout percentage?

So, does increasing Dropout decrease computation time?

Assuming we have a network:

$$L_2 = \sigma(Drop(\textrm{ReLU}(X^{T} \cdot W +b))^{T} \cdot W + b_2)$$,

does e.g., Dropout= 0.2 mean slower computation than Dropout=0.99?

Thus, is the multiplication sparse or remains dense and as such offers no speedups?

Thanks!

2 Answers

This would depend on the exact implementation. While it's possible you could implement some sparse operations to speed up computation when the fraction of dropped out neurons is high, I doubt this is done in practice. Also dropout is a really computationally inexpensive layer (compared to say, a fully connected later), so you may be looking in the wrong area in terms of optimization.

• I see. In fact, I was thinking of reducing the fully connected load via dropout this way, yet it seems this is practically not the case. Thanks! Jul 11 '19 at 4:26

In addition to @shimao's answer, the dropout is typically implemented via masking with 0 which results in additional vector multiplications. It doesn't decrease the complexity, rather increase it slightly, without changing the order of magnitude. For example, it is implemented this way in tensorflow. In PyTorch, I couldn't spot the exact location of the source code, however noting that TF doesn't implement sparse multiplication gives me a hint about how PyTorch would implement it since they're both quality libraries.