There is a statement in this quora answer:

Layer depth is usually a power of 2 because it is convenient for the GPU.

Also, in fully connected layers number of neurons in every hidden layer corresponds to a power of 2.

But, why is the power of 2 convenient for GPU? If I have everything for the power of 3, why would it be inconvenient for GPU? I thought one would use GPU in deep learning only because one can parallelize batches/or convolution computation.

  • 1
    $\begingroup$ Is your computer a binary computer or a ternary one? $\endgroup$ – whuber Dec 7 '18 at 23:05
  • $\begingroup$ How do I understand it? Each GPU will have 2 or 3 cores? $\endgroup$ – Alina Dec 7 '18 at 23:19
  • $\begingroup$ It's more basic than that: ultimately, data are represented on your computer in a unit like a bit. How many states do the bits of your computer have? Two? Three? Ten? $\endgroup$ – whuber Dec 8 '18 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.