# Maximum weight value for a neural network

I am a beginner in the world of neural networks and I have a basic question that I'd like to get the answer to. Is there a way to estimate the maximum value of a weight or a way to keep it in an established range?

ORIGIN OF THE PROBLEM

For now I've only been messing around with neural networks with simple problems (logic gates: xor, and, or, with 2 and more inputs). And what I want to do is to transfer a neural network from a PC to an FPGA.

So far, I've been able to convert the inputs for most problems to value between 0 and 1, which is simple, so I'd like to know if there is a way to estimate the maximum value a weight can have in a specific neural network.

THE PROBLEM

The problem with the size comes from the FPGA's multipliers, which are all 18bits (in my case) for a single multiplier and are not very limited in quantity, but are also not abundant. I know I can increase the resolution by using more than 1 multiplier per multiplication, but it would cut my resources in half and if I dare use floating point, I'd be cutting my resources by 1/5, which is bad for problems with many neurons. For example: My FPGA has 220 multipliers, which would allow for a maximum of 44 neurons per neural network (IF I don't want to use multipliers for the sigmoid function, which is possible in some cases) and I want to execute a neural network that has 64 neurons (real network that I actually would like to execute), so floating point is out of the question in this case.

Imagine now that the maximum weight value is about 60,000, then 16 bits are necessary to represent it, and 2 bits for the decimal points (fixed point arithmetic), which may be bad, depending on the system's needs (I think). Imagine a system where a value like 18.12 wold be translated to 18.00 or 18.25. There is a chance that a problem may have a solution in the PC but not in the FPGA.

So that's my problem. I hope I've been able to give a clear description of the problem so that somebody can help me.

You can keep an eye on a weight during training (just print its value) to get a feel for its magnitude.

If you want a mechanism to keep your weights low, you can add L2 regularization. This simply means you add the sum of the squares of your weights to your loss function, which penalizes the network for high magnitude weights.

That said, in my experience, neural nets tend to keep their weights small. If you have (more or less) zero centered input, and a typical output function, I would be very surprised if the network somehow learned particularily high weights. I would also be surprised if you saw in any noticeable decrease in accuracy from the decreased bit depth. Some people have formally investigated using very low bit depths, for example:

Training deep neural networks with low precision multiplications

"We find that very low precision is sufficient not just for running trained networks but also for training them. For example, it is possible to train Maxout networks with 10 bits multiplications."

Deep Learning with Limited Numerical Precision

"Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy."

• I was doing something like that. I trained the network several times and just averaged the maximum weight, so that I could simulate it later. It worked, but it was too slow. I didn't know about the L2 regularization. About the first quote .. it's true, my tests indicate that with just 5 bits to represent the fractional part is enough to get decent results, and i didn't need more than 4 bits in most cases for the weights Feb 19, 2016 at 21:05
• In most cases with just 8 bits (5 bits for the fractional part and 3 for the integer part) was enough comparing it to the floating point number representation. The thing is that if you use fixed point arithmetic, you need more time to calculate and use more hardware multipliers for just one multiplication. Fixed point multiplication with lower resolution gets me practically the same result, I need only 1 multiplier per operation and can get it done in only one cycle. Nice right? Feb 19, 2016 at 21:18
• Yeah, pretty cool that this works! Feb 20, 2016 at 11:55
• Just to let you know. Yesterday, after the comments I made, I made a few more tests, with more neurons (xor gate) and I noticed something. The result obtained depends a lot on the number of neurons. For example: when I tried the xor gate problem with 3, 4 and 5 neurons, the result was practically identical, but when I tried it with 8 neurons, the rounding used accumulated a lot of error and the end result wasn't as precise as expected. The expectd result for all cases were 0.95 and 0.05 roughly. With 8 neurons the result was 0.93 Feb 20, 2016 at 15:39
• Then I modified the resolution to 16 bits (12bits for the fractional part) and I noticed that all the results were perfect. I'll try to test more networks with different problems and more neurons. But I suspect that this resolution is more than enough for all/most problems. Maybe if the sigmoid function was a mathematical function instead of a quantized table the result will work for 8 bits. That's the next step of implementation Feb 20, 2016 at 15:43

As others pointed out, L2 regularization will help you control the weight growth. I just want to add a couple more points:

1. You should care not about the max values of weights, but about the weight distribution shape. Usually the weights form a Gaussian-like distribution, and if the tails are shallow, you can clip a lot of the tail without affecting accuracy. For example, the values might range from -4 to 4, but 90% of weights will be in the -2 to 2 range. In this case, it should be safe to clip all weights larger than 2, or less than -2. Plot the histogram of the weights to see what I mean.

2. The minimum precision of your weights depends on the number of neurons. The more neurons you have in your network, the less precision you need. If you have enough neurons, you only need binary values everywhere. See this paper for more details: http://arxiv.org/abs/1602.02830

• This is really helpful. I didn't know that about the distribution of weights and the minimum precision required according to size. This was a really big problem for me because it was all developed in VHDL and the available resources I had were limited (LUTs, DSP slices, etc). I needed to implement it in the most economical way possible in order to be capable of implementing medium/large networks. I'll certainly take a closer look on the second point, it has the pontential of being a game changer for my application. Aug 5, 2016 at 16:57
• As an aside, I tried NN on a FPGA a while ago ( github.com/michaelklachko/MLP_FPGA ), and I came to the conclusion that FPGA is not a good platform for the task. FPGA will almost always lose to a GPU if you consider ops/Watt/dollar metric. I recommend you try Nvidia Tegra TX1 board instead, with FP16 precision, or the new Pascal cards with INT8 precision. Aug 5, 2016 at 23:59
• I actually came to a slightly different conclusion. The throughput that I got from the FPGA was much greater than any of my other tests and it allowed me to create a handheld device that can be taken to the field. Right here you could argue for raspberry pi, but the FPGA implementation was much faster. The problems I found came when I tried to implement large networks. With large networks you tend to end all your resources rather quickly, I'm trying to solve this making something a bit slower, but that is capable of implementing larger networks. Aug 7, 2016 at 13:29
• But yeah .. going by Watt/dollar metrics you are completely right. They are expensive power consuming bastards. I'm gonna take a look at your github, thnk you for posting it Aug 7, 2016 at 13:31