least amount of bits needed for single neuron

Question

I wonder what is the smallest amount of bits a neuron needs?

For example for speech recognition / image recognition tasks?

Can 4 bits produce as good results as 16 bits, or even 64 bits ?

Is it not a waste to use 16 bits ( or 64 bit double-s)?

For example spin systems can be encoded with only 1 bit per spin and there is a similarity between spins and neurons (Hopfield network).

Answer 1

The following papers have studied this question (descending chronological order):

• Accelerating Deep Convolutional Networks using low-precision and sparsity. Ganesh Venkatesh, Eriko Nurvitadhi, Debbie Marr. 2016-10-02. https://arxiv.org/abs/1610.00324
• Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1 Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio arxiv: http://arxiv.org/abs/1602.02830
• Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan Deep Learning with Limited Numerical Precision https://arxiv.org/abs/1502.02551
• Courbariaux, Matthieu, Jean-Pierre David, and Yoshua Bengio. "Training deep neural networks with low precision multiplications." arXiv preprint arXiv:1412.7024 (2014). https://arxiv.org/abs/1412.7024
• Vanhoucke, Vincent, Andrew Senior, and Mark Z. Mao. "Improving the speed of neural networks on CPUs." (2011). https://scholar.google.com/scholar?cluster=14667574137314459294&hl=en&as_sdt=0,22 ; https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf

Example from Deep Learning with Limited Numerical Precision:

---

FYI:

• FP16 performance on GTX 1080 is artificially limited to 1/64th the FP32 rate (devtalk.nvidia.com)