# Vector/Matrix sizes in Neural Networks

I'm trying to build my own neural network class (I know there's 1001 already out there, but this is how I learn best). I'm having a bit of trouble with the sizes of certain vectors. For this I'm working off the following source: Neural Networks.

My problem is calculating the z vector (z is the weighted sum of inputs calculated for each neuron in each layer).

z(2) = w(1) * x + b

Where:

w = weight matrix of size (m x n) where m = the number of neuron in the input layer, and n = the number of neuron in second layer
x = input matrix of size (s x t) where s is the number of neuron in the input layer and t is the number of training examples.
b = input matrix of size (c x d) where c is the number of neurons in the second layer and d is the number of neurons in the input layer


This equation changes for the next layer, but that's not something I'm worried about atm. I know that matrix b is essentially just a vector that's been copied across the width.

Now there's a couple of things I can state a fact from the above: m = s = d, n = c. However when I try and process the above equation none of my matrix sizes match.

Can someone tell me given the list of knowns below what the sizes of the matrix z, w, x and b should be.

1. Input size (number of neurons)
2. Number of neurons in layer 2
3. Number of training examples


I've attempted this sort of thing before and I think I always crash and burn when it comes to initialising the arrays (I'm actually using jagged arrays, but they are rectangular).

You say

"w = weight matrix of size (m x n) where m = the number of neuron in the input layer, and n = the number of neuron in second layer"

You should think in terms of "out = func(in)", so "$m$ in, $n$ out", and "$x$ in, $z$ out".

Now you want "out = matrix * in", so the way matrix multiplication works you will need $$n = [n,m] * m$$ i.e. the rule is that adjacent dimensions must be the same in a multiplication formula.

So something seems off. Indeed, if you look at the first equation in your link, it is the transpose of the weight matrix that is used $$z=W^Tx$$ This makes sense, because $[m,n]^T=[n,m]$, as required.

• you haven't answered it.. – Euler_Salter Oct 20 '17 at 9:37

Chuck Anderson does a great job of explaining these details here:

http://nbviewer.jupyter.org/url/www.cs.colostate.edu/~anderson/cs480/notebooks/10%20Nonlinear%20Regression%20with%20Neural%20Networks.ipynb

The associated video is also very good. HTH