4
$\begingroup$

I'm trying to find a way to estimate the number of weights in a neural network. Let's look a simple example.

nnetar(1:10) (from the forecast package in R) gives me a 1-1-1 Network with 4 weights.

enter image description here

That makes total sense to me, since we see four arrows in the illustration.

However, nnetar(1:10, xreg=data.frame(10:1,3:12)) gives me a 3-2-1 Network with 11 weights

enter image description here

I don't understand, why the output says that there are 11 weights involved, since I count 12!? Any suggestions?

$\endgroup$
  • $\begingroup$ The topic of the question seems to be neural networks yet the tool that you're using makes no sense outside time series analysis. Function nnetar is a wrapper of nnet (from package caret) which adapts autoregressive neural networks to time series forecasting. Are you sure this is what you want? $\endgroup$ – Digio Aug 9 '17 at 9:12
  • 1
    $\begingroup$ Also, in any type of neural network, the weights are between the input layer and the hidden layers, between hidden layers, and between hidden layers and the output layer. There are no weights outside the input and output layer. In your two figures I'm only seeing 2 and 8 weights, respectively. $\endgroup$ – Digio Aug 9 '17 at 9:16
  • $\begingroup$ Hi Digio, thanks for the reply. I'm using nnetar in a time series context, where it produces good forecasts. So, yes nnetar is want I want. I just gave this example for demonstration purposes. $\endgroup$ – shb Aug 14 '17 at 11:34
7
$\begingroup$

The reason you're confused is the fact that function nnetar creates an autoregressive neural network and not a standard neural network. This means that the input layer nodes of the network are:

  • the exogenous regressors that you pass with xreg ,
  • the autoregressive variables that nnetar creates,
  • the bias term.

Running nnetar(1:10, xreg=data.frame(10:1,3:12)), for example, creates by default a NNAR(1,2) model, i.e. a neural network with one lagged term and two hidden nodes.

NNAR(1,2) with two regressors results to a 3-2-1 network where you have:

  • 3 nodes in the input layer: $y_{t-1}$, $x_1$, $x_2$
  • 2 nodes in the hidden layer
  • 1 node in the output layer

If you calculate all weights so far you'll see that you only get 8: $3 \times 2 + 2 \times 1 $. But then why does nnetar return 11 weights? This is because of the "bias" nodes, which are not really counted in the 3-2-1 network though they are part of it and do carry extra weights. There is one bias node in the input layer and one in the hidden layer which connects only to the output layer. So you have 2 weights from the input layer bias node plus 1 weight from the hidden layer bias node, that makes 3 plus 8 from before, 11 weights in total. You can learn more on this architecture from the documentation of nnetar or Hyndman's new book.

Here's what NNAR(1,2) with 2 regressors looks like:

NNAR(1,2), 2 regressors: 3-2-1 network, 11 weights

You can find the number of weights by counting the edges in that network.

To address the original question:

In a canonical neural network, the weights go on the edges between the input layer and the hidden layers, between all hidden layers, and between hidden layers and the output layer.

If you are looking for a way to count weights in a 1-hidden-layer network that would be the number of nodes in the hidden layer times number of nodes in the input layer plus number of nodes in the hidden layer times number of nodes in the output layer. If you're using nnetar, you must make sure you add the autoregressive terms as nodes in the input layer (nnetar that always comes with 1 hidden layer but if you have more hidden layers you simply have to adapt this method to all layers).

$\endgroup$
  • 1
    $\begingroup$ Hi Digio, thanks for the comprehensive answer. I didn't know about the bias nodes, because Hyndman's Book doesn't mention them. Now I understand where the number of weights come from. Great! $\endgroup$ – shb Aug 14 '17 at 11:39
  • $\begingroup$ It's worth noting that if there's no hidden layer, you simply multiply the # of nodes in the input layer by the number of nodes in the output layer to calc the # of total weights. $\endgroup$ – logankilpatrick Jul 29 '20 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.