Maximum weight value for a neural network I am a beginner in the world of neural networks and I have a basic question that I'd like to get the answer to. Is there a way to estimate the maximum value of a weight or a way to keep it in an established range?
ORIGIN OF THE PROBLEM
For now I've only been messing around with neural networks with simple problems (logic gates: xor, and, or, with 2 and more inputs). And what I want to do is to transfer a neural network from a PC to an FPGA. 
So far, I've been able to convert the inputs for most problems to value between 0 and 1, which is simple, so I'd like to know if there is a way to estimate the maximum value a weight can have in a specific neural network. 
THE PROBLEM
The problem with the size comes from the FPGA's multipliers, which are all 18bits (in my case) for a single multiplier and are not very limited in quantity, but are also not abundant. I know I can increase the resolution by using more than 1 multiplier per multiplication, but it would cut my resources in half and if I dare use floating point, I'd be cutting my resources by 1/5, which is bad for problems with many neurons. For example: My FPGA has 220 multipliers, which would allow for a maximum of 44 neurons per neural network (IF I don't want to use multipliers for the sigmoid function, which is possible in some cases) and I want to execute a neural network that has 64 neurons (real network that I actually would like to execute), so floating point is out of the question in this case. 
Imagine now that the maximum weight value is about 60,000, then 16 bits are necessary to represent it, and 2 bits for the decimal points (fixed point arithmetic), which may be bad, depending on the system's needs (I think). 
Imagine a system where  a value like 18.12 wold be translated to 18.00 or 18.25. There is a chance that a problem may have a solution in the PC but not in the FPGA.
So that's my problem. I hope I've been able to give a clear description of the problem so that somebody can help me. 
 A: You can keep an eye on a weight during training (just print its value) to get a feel for its magnitude.
If you want a mechanism to keep your weights low, you can add L2 regularization. This simply means you add the sum of the squares of your weights to your loss function, which penalizes the network for high magnitude weights.
That said, in my experience, neural nets tend to keep their weights small. If you have (more or less) zero centered input, and a typical output function, I would be very surprised if the network somehow learned particularily high weights. I would also be surprised if you saw in any noticeable decrease in accuracy from the decreased bit depth. Some people have formally investigated using very low bit depths, for example:
Training deep neural networks with low precision multiplications
"We find that very low precision is sufficient not just for running trained networks but also for training them. For example, it is possible to train Maxout networks with 10 bits multiplications."
Deep Learning with Limited Numerical Precision
"Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy."
A: As others pointed out, L2 regularization will help you control the weight growth. I just want to add a couple more points: 


*

*You should care not about the max values of weights, but about the weight distribution shape. Usually the weights form a Gaussian-like distribution, and if the tails are shallow, you can clip a lot of the tail without affecting accuracy. For example, the values might range from -4 to 4, but 90% of weights will be in the -2 to 2 range. In this case, it should be safe to clip all weights larger than 2, or less than -2. Plot the histogram of the weights to see what I mean.

*The minimum precision of your weights depends on the number of neurons. The more neurons you have in your network, the less precision you need. If you have enough neurons, you only need binary values everywhere. 
See this paper for more details: http://arxiv.org/abs/1602.02830
