Computation time with respect to Dropout

Question

I've been recently attempting to speed up neural network training (in PyTorch). My question is the following.

Does the computation time of a given feedforward neural network vary based on Dropout percentage?

So, does increasing Dropout decrease computation time?

Assuming we have a network:

$L_2 = \sigma(Drop(\textrm{ReLU}(X^{T} \cdot W +b))^{T} \cdot W + b_2)$,

does e.g., Dropout= 0.2 mean slower computation than Dropout=0.99?

Thus, is the multiplication sparse or remains dense and as such offers no speedups?

Thanks!

Answer 1

This would depend on the exact implementation. While it's possible you could implement some sparse operations to speed up computation when the fraction of dropped out neurons is high, I doubt this is done in practice. Also dropout is a really computationally inexpensive layer (compared to say, a fully connected later), so you may be looking in the wrong area in terms of optimization.

Answer 2

In addition to @shimao's answer, the dropout is typically implemented via masking with 0 which results in additional vector multiplications. It doesn't decrease the complexity, rather increase it slightly, without changing the order of magnitude. For example, it is implemented this way in tensorflow. In PyTorch, I couldn't spot the exact location of the source code, however noting that TF doesn't implement sparse multiplication gives me a hint about how PyTorch would implement it since they're both quality libraries.