6
$\begingroup$

Could someone provide just a quick overview on what I am seeing when I see a typical example of a neural network and how it relates to the matrix math behind it? We normally see a typical graph like the below:

enter image description here

Now, according to the author, in matrix math, this is represented in the below, where Ws are the coefficients to whatever activation function is in a node of hidden layer 1 and X are the features or predictor values: enter image description here

When it comes to estimating the parameters of each hidden layer, this representation makes sense. We have a bunch of data, and we feed them all into the model and produce predictions. If we have 5 observations, we have 5 predictions in the picture. Thus, the number of inputs equals the number of outputs.

What I don't really understand is once the model is built, how does the graphical representation work? Let's say you have a single new predictor value and you feed it into the NN below. How does NN know which coefficient is assigned to this single predictor value? What happens to the other coefficients? Would the vector of Xs values be equal to [X1, 0, 0, 0, 0]?

Thanks.

$\endgroup$

2 Answers 2

5
$\begingroup$

I believe you've misunderstood the role of input variables (or I've misunderstood you). $X_1...X_5$ are your features belonging to a single observation in the data. So, input dimension is $5$, meaning your data is five dimensional. This has nothing to do with the output dimension, which is typically equal to the number of classes in your data. So, when a new data sample comes, you just feed its $X_1..X_5$ to the network and get your output.

$\endgroup$
5
  • 2
    $\begingroup$ Oh I see. I was googling some other pictures and the output layer has a different number of values than the input layer so I got a bit confused. This makes sense. $\endgroup$
    – user38283
    Commented Jul 26, 2020 at 20:13
  • $\begingroup$ So for something like this: assets.bwbx.io/images/users/iqjWHBFdfxIU/ibOkMAUoy_Wc/v0/…, we have 3 features, each node in the first green layer has a coefficient matrix that is 5x3, each node in the second green layer has a coefficient matrix that is 1x5, and the black node has a coefficient matrix that is 1x5 as well to produce a single predicted value? $\endgroup$
    – user38283
    Commented Jul 26, 2020 at 20:16
  • 1
    $\begingroup$ In the first green layer each node has coefficient matrix of size $1\times3$, since there are $5$ nodes, it makes a $5\times3$ coef. matrix for the layer, excluding bias. $\endgroup$
    – gunes
    Commented Jul 26, 2020 at 20:20
  • $\begingroup$ Got it. So each node in the first green layer produces a single value that is fed into the nodes of the second green layer, as opposed to each node in the first green layer producing 5 different values that are fed into the 5 nodes of the second green layer. Thanks! $\endgroup$
    – user38283
    Commented Jul 26, 2020 at 20:25
  • 1
    $\begingroup$ That is correct. $\endgroup$
    – gunes
    Commented Jul 26, 2020 at 20:26
1
$\begingroup$

Just for clarity I will refer to the image of your question. From left to right, this Neural Network (NN) has one input layer of 5 neurons, one hidden layer of 2 neurons and one output layer of 5 neurons.

how does the graphical representation work? Let's say you have a single new predictor value and you feed it into the NN below. How does NN know which coefficient is assigned to this single predictor value? What happens to the other coefficients? Would the vector of Xs values be equal to [X1, 0, 0, 0, 0]?

Once we have already set up all the weights and biases the Neural Network has been already trained. Thereby the net is ready to receive data in its input layer and make the correct prediction at the output layer.

In the NN of the image, we have an input layer formed by 5 neurons. This 5 neurons will ''receive'' the data whose values are known e.g. they could be a 5x1 vector of pixel gray scale values of an image formed by 5 pixels (the same as the number of neurons of the input layer).

After this, the network will do all the mathematical operations needed and will give as a result, following the example of the image, a 5x1 vector at the output layer. This output could give some information of it e.g. if it contains 1 of a total of 5 objects, so if the 1st neuron of the output layer is asociated with object1, this neuron will be activated meaning that this object is present.

Following this, if object1, object3 and object4 are present in the image, we will have at the output layer a vector like this: $[1, 0, 1, 1, 0]^T$. Note: In practice, this will happen only ideally. If the network is well trained, the most likely thing to happen is that the vector will have elements close to $0$ for the non-activated neurons, and close to $1$ for the activated neurons.

and how it relates to the matrix math behind it?

In order to answer this question I am going to need some notation. As the Network is already trained, we know all the elements of the matrices of the image: Each weight $w_{jk}$ and each bias $b_{j}$ where $j$ represents the row of the weight matrix (or vector of biases) and it is also related to the neuron position in the layer from we want to calculate the vector of activations. On the other hand, $k$ represents the column of the weight matrix or the position of the neuron in the previous layer of the current one in wich we are calculating the activations.

With this notation we can calculate the activations of each neuron $j$ in a layer as:

$x_j = f(z_j)\,\,$ with $\,\,z_j = \sum_k w_{jk}\times x_k + b_j$ where $f$ represents the activation function, this can be of different kinds like Sigmoid or Softmax.

Note that this is the same as calculating separately each $Z_i$ of the equation that appears in the image of the question. So this is what the matrix math do "behind the scenes".

Edit: As Gunes has correctly say, the number of neurons at the input layer has nothing to do with the number of neurons at the output layer.

$\endgroup$
1
  • $\begingroup$ Thanks for the in depth explanation. $\endgroup$
    – user38283
    Commented Jul 27, 2020 at 1:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.