least amount of bits needed for single neuron I wonder what is the smallest amount of bits a neuron needs? 
For example for speech recognition / image recognition tasks?
Can 4 bits produce as good results as 16 bits, or even 64 bits ?
Is it not a waste to use 16 bits ( or 64 bit double-s)?
For example spin systems can be encoded with only 1 bit per spin and there is a similarity between spins and neurons (Hopfield network).
 A: The following papers have studied this question (descending chronological order):


*

*Accelerating Deep Convolutional Networks using low-precision and sparsity. Ganesh Venkatesh, Eriko Nurvitadhi, Debbie Marr. 2016-10-02. https://arxiv.org/abs/1610.00324

*Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio
arxiv: http://arxiv.org/abs/1602.02830

*Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan
Deep Learning with Limited Numerical Precision https://arxiv.org/abs/1502.02551

*Courbariaux, Matthieu, Jean-Pierre David, and Yoshua Bengio. "Training deep neural networks with low precision multiplications." arXiv preprint arXiv:1412.7024 (2014). https://arxiv.org/abs/1412.7024

*Vanhoucke, Vincent, Andrew Senior, and Mark Z. Mao. "Improving the speed of neural networks on CPUs." (2011). https://scholar.google.com/scholar?cluster=14667574137314459294&hl=en&as_sdt=0,22 ; https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf
Example from Deep Learning with Limited Numerical Precision:


FYI:


*

*FP16 performance on GTX 1080 is artificially limited to 1/64th the FP32 rate (devtalk.nvidia.com)
