Formula for number of weights in neural network

Question

I'm trying to find a way to estimate the number of weights in a neural network. Let's look a simple example.

nnetar(1:10) (from the forecast package in R) gives me a 1-1-1 Network with 4 weights.

That makes total sense to me, since we see four arrows in the illustration.

However, nnetar(1:10, xreg=data.frame(10:1,3:12)) gives me a 3-2-1 Network with 11 weights

I don't understand, why the output says that there are 11 weights involved, since I count 12!? Any suggestions?

Answer 1

The reason you're confused is the fact that function nnetar creates an autoregressive neural network and not a standard neural network. This means that the input layer nodes of the network are:

• the exogenous regressors that you pass with xreg ,
• the autoregressive variables that nnetar creates,
• the bias term.

Running nnetar(1:10, xreg=data.frame(10:1,3:12)), for example, creates by default a NNAR(1,2) model, i.e. a neural network with one lagged term and two hidden nodes.

NNAR(1,2) with two regressors results to a 3-2-1 network where you have:

• 3 nodes in the input layer: $y_{t-1}$, $x_1$, $x_2$
• 2 nodes in the hidden layer
• 1 node in the output layer

If you calculate all weights so far you'll see that you only get 8: $3 \times 2 + 2 \times 1 $. But then why does nnetar return 11 weights? This is because of the "bias" nodes, which are not really counted in the 3-2-1 network though they are part of it and do carry extra weights. There is one bias node in the input layer and one in the hidden layer which connects only to the output layer. So you have 2 weights from the input layer bias node plus 1 weight from the hidden layer bias node, that makes 3 plus 8 from before, 11 weights in total. You can learn more on this architecture from the documentation of nnetar or Hyndman's new book.

Here's what NNAR(1,2) with 2 regressors looks like:

You can find the number of weights by counting the edges in that network.

To address the original question:

In a canonical neural network, the weights go on the edges between the input layer and the hidden layers, between all hidden layers, and between hidden layers and the output layer.

If you are looking for a way to count weights in a 1-hidden-layer network that would be the number of nodes in the hidden layer times number of nodes in the input layer plus number of nodes in the hidden layer times number of nodes in the output layer. If you're using nnetar, you must make sure you add the autoregressive terms as nodes in the input layer (nnetar that always comes with 1 hidden layer but if you have more hidden layers you simply have to adapt this method to all layers).